Weight-level pruning

[msakarvadia]

• [ ] Task vectors:

train one model on only noise (this is your task vector)

train one model on clean+noise (model w/ memorization)

to mitigate memorization subtract your task vector from model w/ memorization

• [x] Maini greedy search based localization
• [ ] Lottery ticket localization
• [x] Random network finding sparse sub-networks localization
• [x] Neuro-toxin
• [x] OBS: second order pruning (https://arxiv.org/pdf/2203.07259)

Maybe invent my own technique which is a mix between greedy iterative pruning and OBS:

• [x] Greedy OBS

[msakarvadia]

• [x] Neuro-toxin is implemented, but want to separate out full model localization and per-layer localization
• [x] implement ratios for greedy search

[msakarvadia]

• [x] Greedy: restrict to MLPs

[msakarvadia]

This current repo has all of the weight-level methods we are going to support for now.