search

RAIVNLab / supsup

Code for "Supermasks in Superposition"
117 stars 19 forks source link

3.4 NNs Scenario #4

Open bhack opened 4 years ago

bhack commented 4 years ago

I have a question about the NNs scenario in 3.4. An hypotesis about feeding specifical adversarial examples that could go to saturate the network capacity generating a proliferation of new supermasks tasks or simpler supermasks proliferation by more or less long tail samples in a continous learning setup. Have you never thought about exploring a solution for supermask expansion e.g. extending the Supermask with more resoruces/connection over the same task?
Have you never explored about assign temporary supermasks and have a "consolidation phase" where new supermasks could be "merged" with previous supermasks or promoted as a new (Independent) task/Mask allocation?

mitchellnw commented 4 years ago

That is a good idea, no we have not thought about this! It is difficult as supermasks which do the same thing could look very different.

bhack commented 4 years ago

Yes It Is a not so much explored topic. I found some early step about reasoning on masks simliarity in Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding but It Is not strictly what we are talking about here.