search

pytorch / captum

Model interpretability and understanding for PyTorch
https://captum.ai
BSD 3-Clause "New" or "Revised" License
4.8k stars 485 forks source link

Plan for perturbation-based methods #46

Closed marcoancona closed 4 years ago

marcoancona commented 4 years ago

Hello, Kudos for the great work. I believe this has great potential. I wonder what is in your roadmap, especially regarding perturbation-based attribution methods (Occlusion, LIME/KernelSHAP, Shapley Value sampling, etc.).

Are these planned at all? While being orders of magnitude slower, these methods have the advantage that they can be applied to any black-box model (ie. any network architecture is supported out-of-the-box, with no need to instrument layers or implement custom modules). The implementation into Captum should be easier too. Moreover, Shapley Value attributions have unique theoretical properties that might be important when speed is not critical.

While it makes sense to focus on gradient-based methods first, maybe the structure of the library should be such that these methods can be easily added in the future.

NarineK commented 4 years ago

Hi Marco,

thank you so much for your feedback! We really appreciate that. Yes, perturbation-based methods are definitely planned. Originally, we were thinking to add them in the first release but decided to postpone and focus on the gradient based approaches first. The library should be easily extensible for perturbation based approaches as well.

Thank you

orionr commented 4 years ago

@NarineK and @vivekmig are actively working on this. Closing, but we can reopen later if needed. Thank you for the question!