Model explainability that works seamlessly with 🤗 transformers. Explain your transformers model in just 2 lines of code.
Apache License 2.0
1.27k
stars
96
forks
source link
How to interpret the model fine tuning on the pre-trained ViT model using the imagery with larger resolution (500 * 500) than the pre-trained dataset (224 * 224) #114
I am working on the downstream task of image classification. I fine tuned a pre-trained ViT model (224 224) using larger resoltuion imagery (500 500). In the process of fine tuning, I only need to set interpolate_pos_encoding as True in model(inputs, interpolate_pos_encoding = True) to fine tune the pre-trained model. However, it seems like your code does not include the part of interpolate_pos_encoding in the class of ImageClassificationExplainer*. I am wondering what do you think about the attention map of the image classified by my fine tuned model according to your understanding to ViT. Is it much different from the attention map of the image classified by a model fine tuned from the pre-trained ViT model using the same larger resolution imagery? (In this process of fine tuning, imagery is resized to 224 224.) Or they would be same?
Hi @cdpierse ,
I am working on the downstream task of image classification. I fine tuned a pre-trained ViT model (224 224) using larger resoltuion imagery (500 500). In the process of fine tuning, I only need to set interpolate_pos_encoding as True in model(inputs, interpolate_pos_encoding = True) to fine tune the pre-trained model. However, it seems like your code does not include the part of interpolate_pos_encoding in the class of ImageClassificationExplainer*. I am wondering what do you think about the attention map of the image classified by my fine tuned model according to your understanding to ViT. Is it much different from the attention map of the image classified by a model fine tuned from the pre-trained ViT model using the same larger resolution imagery? (In this process of fine tuning, imagery is resized to 224 224.) Or they would be same?
Thanks, Yin