model.feature_list function in OOD_Generate_Mahalanobis.py

[samunaai]

Dear Andre, Greetings. Hoping you are doing well! I was hoping to ask one question about the model definitions used in mahalanobis_and_odin/OOD_Generate_Mahalanobis.py. I noticed that in line 83, we are using a function called feature_list via temp_list = model.feature_list(temp_x)[1]? However, this method is not currently defined in the models in my_models, such as my_models.resnet_50, I was hoping to inquire regarding how you were able to define these functions? Just so I could maintain compatibility with the versions of PyTorch you used in torchvision.models. Thank you so much! Warm regards,

[paaatcha]

Hello @munachisonwadike. Sorry for the late reply. I was investigating your problem and I identified the reason. In the email you have sent to us, Chamdramoulli has shared with the original repository. The authors have coded the models in a different way. They've changed its structure to capture the features. You may find the feature_list method here.

The way we do it in this repository is different. We set a hook to get the features of interest. You may see it here in the Setting the hook registercell. That's why you cannot find the method. Particularly I think using the hooker is better since we don't modify anything in the original model.

A quick solution: you may define the hook function similar to the one here and capture the features. Another solution would be copying the models from the original repository.

Can you check it?

[samunaai]

Fantastic let me try

[samunaai]

Dear Andre @paaatcha, Greetings! I hope this message finds you well. Thank you so very much for your feedback. I have implemented some basic technique to extract the features from the same 5 layers of resnet as in the original implementation. This can be found in my fork of the resnet_50.py here. I borrowed the isinstance() function used in the hooks, but I directly take those feature layers which were of interest, in the same way as was done in the original repo with the feature_list and intermediate_forward functions (the first convolution layer after ReLU, and the other 4 resnet block layers) However, I noticed that I am getting slightly different results from what was reported in your paper for the unbiased evaluation of resnet_50 using TNR (Mahalanobis). I attached the screenshots of the results below. Would the results still be within the margin of error? Also, I noticed that only TNR metric was provided in the paper for unbiased evaluation. Do the results from OOD_Regression_Mahalanobis.py correspond to the unbiased evaluation scores? Thank you so much once again, and I look forward to hearing from you. Warm regards,

[samunaai]

Greetings @paaatcha! I hope this message finds you well. Just a small addition: I also made an implementation of the resnet_50.py where I used hooks instead of directly taking the features. However when I try to use hooks as was done in the sk_resnet-50.ipynb, the hook function is using the criterion if isinstance(layer, models.resnet.Bottleneck). However, this will return 16 layers of resnet, instead of just 5 layers that was done in the original implementation. The problem is that 16 layers becomes too heavy for the sample_estimator function in lib_generation.py, since it performs a lot of heavy matrix computations. This results in an OOM error on our relatively small GPU (screenshot below). Did you use a different isinstance criterion for your gram-ood and mahalanobis implementations? (i.e a different set of layers from which we hook the features?) Perhaps something closer to 5 features in total rather than 16 used in gram? Thank you so much once again, and I look forward to hearing from you. Warm regards,

[chandramouli-sastry]

The mahalanobis algorithm computes covariance matrices and that's costly: using features only at the end of the resnet/densenet blocks works best for mahalanobis.

On Mon., Oct. 18, 2021, 7:22 p.m. Munachiso, @.***> wrote:

I was also hoping to add another vital point. that I made an implementation of the resnet_50.py where I used hooks instead of directly taking the features. However when I try to use hooks as was done in the sk_resnet-50.ipynb https://github.com/munachisonwadike/gram-ood/blob/master/sk_resnet-50.ipynb, the hook function is using the criterion if isinstance(layer, models.resnet.Bottleneck). However, this will return 16 layers of resnet, instead of just 5 layers, as was done in the original implementation https://github.com/pokaxpoka/deep_Mahalanobis_detector/blob/master/models/resnet.py. The problem is that 16 layers becomes too heavy for the sample_estimator function in lib_generation.py, since it performs a lot of heavy matrix computations. This results in an OOM error on our relatively small GPU (screenshot below). Did you use a different isinstance criterion for your gram-ood and mahalanobis implementations? (i.e a different set of layers from which we use the features?) Perhaps something closer to 5 features in total rather than 16 used in gram? Thank you so much once again, and I look forward to hearing from you. Warm regards,

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/paaatcha/gram-ood/issues/2#issuecomment-946215134, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ2SNBNVLOGVWDLGW5YPMLUHSMZTANCNFSM5FULQETA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[paaatcha]

Greetings @paaatcha! I hope this message finds you well. Just a small addition: I also made an implementation of the resnet_50.py where I used hooks instead of directly taking the features. However when I try to use hooks as was done in the sk_resnet-50.ipynb, the hook function is using the criterion if isinstance(layer, models.resnet.Bottleneck). However, this will return 16 layers of resnet, instead of just 5 layers that was done in the original implementation. The problem is that 16 layers becomes too heavy for the sample_estimator function in lib_generation.py, since it performs a lot of heavy matrix computations. This results in an OOM error on our relatively small GPU (screenshot below). Did you use a different isinstance criterion for your gram-ood and mahalanobis implementations? (i.e a different set of layers from which we hook the features?) Perhaps something closer to 5 features in total rather than 16 used in gram? Thank you so much once again, and I look forward to hearing from you. Warm regards,

Hello @munachisonwadike! Sorry for the late reply, I'm super busy this week. I was going to add @chandramouli-sastry to this conversation (he performed the experiments using ODIN and Mahalanobis), but he's already answered you.