AntonioCarta
commented
1 month ago Maybe the easiest way is to put a function here that computes the linear probe accuracy and add an example of plugin that publishes the probing accuracy. I'm guessing that the plugin may need to be adapted by the user to choose when/what to compute, but the probing itself is quite general, it just needs the model, stream and layer name.
I would like to add some linear probing tools since it has been a widely used technique in evaluation of continual learning and non continual learning methods. I don't know what is the best way to do it. I think we could have it as an external layer that we add onto an existing representation, maybe with an attached method that trains it or an external method that trains it. I have to think about it.