Lottery ticket hypothesis

[bhack]

What about the lottery ticket hypothesis and follow-up works like large scale and Uber Supermask? Is there something reletad in the roadmap?

[raziel]

Hi. Indeed there's lost of techniques, which tend to vary on model coverage and ease of use.

Initially we're trying to provide max coverage for models, and use cases, while selecting algorithms that we have validated across models, do not require too much fine tuning, and deliver immediate benefits (e.g. compression and/or performance boost).

You can see our up to date plans at https://www.tensorflow.org/model_optimization/guide/roadmap That's a list of techniques that are either will be available soon, or planning is already in progress (i.e. less likely to change).

Having said that, we're working with internal Google groups to bring more state-of-the-art techniques, as well as will open the repository for external contributions, in the near future. Again, currently the focus is in having a well curated API, rather than many experimental techniques with unproven benefits (by us anyway).

Hope you get to try our APIs, and give us feedback.

Thanks!

[bhack]

I supposed you had something more to say cause internally tested in Google months ago https://github.com/google-research/lottery-ticket-hypothesis

[raziel]

Update: we just did some experiments in some models (cannot share since they're internal, not open source), and in those cases the lottery ticket technique rendered worse results (5-15%) than the technique implemented in this API --actually with some small tweaks to the training script you can implement the lottery ticket principles with our existing API.

I don't like making strong statements on the merits of a technique --i.e. I'm not saying lottery ticket does not work well. Particularly since I cannot share the test methodology (data, steps, etc.) nor models, but just giving a data point. As with many things in ML, variables like data and training steps can results in significant outcomes in results (i.e. maybe lottery ticket is still good for small amount of data/training steps).

[bhack]

Thanks for the covered share. If you can disclose have you tried with Uber Supermask and other follow-ups or just the plain lottery?

[bhack]

Just to mention some recent follow-ups: https://arxiv.org/abs/1903.01611 https://arxiv.org/abs/1907.04840

[bhack]

Just to notify about competitor https://ai.facebook.com/blog/understanding-the-generalization-of-lottery-tickets-in-neural-networks/

[bhack]

New related work: What's Hidden in a Randomly Weighted Neural Network?https://arxiv.org/abs/1911.13299

[bhack]

just to take a look at the competitors.. https://pytorch.org/tutorials/intermediate/pruning_tutorial.html

[raziel]

With the current pruning API you can implement the lottery ticket hypothesis. We have done that internally, and may be releasing a colab as a tutorial.

If you have specific requests for the API to meet a specific need we're welcome to the feedback, and the best way of submitting the request is via a ticket that describes the need and provides an example in code showing the requirement. We also welcome PRs with improvements to the API.

Thanks

[bhack]

Yes if you have a lottery ticket/Supermask colab to add I think that currently could be enough to support this type of development with the library for third party developers.

[drubinstein]

Any updates on this? I would also be interested in a colab.