search

pytorch / captum

Model interpretability and understanding for PyTorch
https://captum.ai
BSD 3-Clause "New" or "Revised" License
4.79k stars 485 forks source link

TCAV for multiple instance learning #840

Open qinghezeng opened 2 years ago

qinghezeng commented 2 years ago

Hi, many thanks for this great work!

I would like to ask does it make sense to use TCAV to test an attention based multiple instance network? If yes, the examples of a concept should be bags or instances?

Thank you very much!

99warriors commented 2 years ago

Very interesting application! It makes sense for examples of a concept to be bags - you can then interpret the bag embedding layer. If the network can make predictions for instances, then it would also make sense for examples of a concept to be instances - as long as you have an instance classifier, TCAV does not care whether the instance classifier was trained using bag labels or instance labels, and you can interpret an instance-specific layer. However, if the network cannot make predictions for instance (i.e. Ilse et al), then having the examples of a concept to be instances is a bit weird, and I need to think a bit more if that makes sense.