Inconsistency prediction on ShearletX representation across classes

[MRAWAY77]

Hi skmda,

I attended your booth 203 on 22/06 during the CVPR23 conference. I have tested your code and it works as per your presentation.I remember you mentioning that the ShearletX representation was being sent for additional inference to verify the output of the classifier.

My surface testing proves the additional inference on the original model for 'tree frog' stands valid.

However, when i begin to randomly select some online images and shown the results below

based on VGG19, the classifier output Suit in this image

With ShearletX representation, the classifier output GasMask on VGG19.

I have a few questions about its usage, and I'm not sure if you're open to discussing it. Let me know if you are open for it and i look forward to your reply.

[skmda37]

Hi! The final prediction can change after applying the shearlet or wavelet mask, however I would be surprised if the original prediction does not stay in the top-5 imagenet classes. There is a trade-off between mask sparsity (deleting image coefficients) and distortion in the model output after masking. For the suit example you see that 28% of the label probability for suit is kept. That is apparently not enough to be the top-1 prediction. But it is probably in the top-5. If you want to have a mask that maintains the final prediction you may want to decrease the hyperparameter controlling the mask sparsity.