search

Lightning-AI / pytorch-lightning

Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.
https://lightning.ai
Apache License 2.0
28.16k stars 3.37k forks source link

Implement a PyTorch Lighning version of the 'Robust Attribution Regularization' paper #505

Closed kzk2000 closed 4 years ago

kzk2000 commented 4 years ago

Is your feature request related to a problem? Please describe. It's a new regularization method which I believe is very intriguing to be part of the larger Pytorch lightning training stack for lots of people to try out:

Summary of the feature is in this blog: https://deep.ghost.io/robust-attribution/ Full paper: https://arxiv.org/abs/1905.09957

Describe the solution you'd like Take @jfc43 's Tensorflow code https://github.com/jfc43/robust-attribution-regularization and translate it to Pytorch Lightning to make it easy to use on any data set. I'm not an low level Tensorflow/Pytorch coder and am asking for help from the PyTorch Lightning community to make this regularization method easy-to-use within the PyTorch Lightning framework.

williamFalcon commented 4 years ago

@jeffling @neggert any thoughts?

neggert commented 4 years ago

I don't think we should be in the business of implementing every new technique that's published. At most, we could provide an example showing how to do it in Lightning.

jeffling commented 4 years ago

Agreed, I think it's outside of the scope of this project. We need to be careful of overfitting to any specific set of techniques as well with how fast this field moves.

If we were to implement any technique, it could be framework-level hooks to make things like transfer learning (seen a few issues about this) easier, or data augmentation (seen a few about this as well) easier.

williamFalcon commented 4 years ago

agreed. maybe the best thing is for @kzk2000 to create an external repo and share the link here? that way people can pip install if they want to use it :)

kzk2000 commented 4 years ago

Thanks for taking a look at this and agreed, such a request likely opens the door to many similar "just published" ones. Let's see how much attention this paper gets at NIPS in the next weeks, and hopefully some savvy Pytorch will jump on it :-)

Appreciate you guys taking a look, regularization methods like this are critical for high stakes environments like finance / asset management (my domain), hence my interest to try it out.