search

kkirchheim / pytorch-ood

👽 Out-of-Distribution Detection with PyTorch
https://pytorch-ood.readthedocs.io/
Apache License 2.0
255 stars 24 forks source link

Does this repo include energy-based detection with margin? #72

Closed amztc34283 closed 2 months ago

amztc34283 commented 2 months ago

https://www.youtube.com/watch?v=Ga09-9JItxs

I saw the repo have energy-based implementation but there was a new paper using additional margin, do you have this implemented?

If not, I am planning to create a PR.

kkirchheim commented 2 months ago

Is this the supervised "Energy Regularization" loss where they enforce an energy gap between IN and OOD data during training? This method was, iirc, proposed in the original energy-based OOD detection paper. If so, it it implemented here. Pretrained ResNet-40 weights from the original repo are also available for CIFAR10+100.

amztc34283 commented 2 months ago

I don't know if that is the case, this is the link to the paper - Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection. If the paper you are referring to is Energy-based out-of-distribution detection (2020), then the paper I mentioned is an extension of that to handle covariate-shifted data.

Looks like the paper you are referring is the paper from 2020, then I think we could extend to this new method.

kkirchheim commented 2 months ago

Okay, I see. No, this is not yet included. If you would like to contribute an implementation, we would highly appreciate it.

amztc34283 commented 2 months ago

How long does the pytest usually take?

Is there a way to run those tests on my MacBook? Maybe I can't if the fine-tuning takes a long time.

kkirchheim commented 2 months ago

The tests should only take a couple of minutes. There is no fine-tuning done, so that should not be an issue. However, some of the tests require network access to download datasets, so if your connection is slow, this might be an issue. Overall, tests should be platform-independent and do not require cuda. Can you narrow it down to a particular test execution?

amztc34283 commented 2 months ago

Let me get back to you this weekend.

amztc34283 commented 2 months ago

Since the energy-based OOD with margin requires fine-tuning on pre-trained model, do you want me to write tests that involves fine-tuning? In general, what tests do you suggest to write for a new detection method?

kkirchheim commented 2 months ago

do you want me to write tests that involves fine-tuning?

No, but the tests should ensure that you can do a forward and a backward pass through the method. Maybe have a look at the existing tests to get some inspiration on what to test.

amztc34283 commented 2 months ago

My repo is almost ready for pull request. I still have to work on some documentation this week and aim to merge by the end of week.

amztc34283 commented 2 months ago

Closing as merge request is created!