The code of opengan is incomplete, missing two important parts

[liujunzhuo]

I ran opengan's code and found poor results. Then, I checked the paper as well as Excel table you published and the results are the same. I read the code and found that it was missing two important parts.

1. Open Validation: The original paper says that using outlier data as a valset to select the “right” GAN-discriminator does achieve the state-of-the-art on open-set discrimination. In other word, without open validation, the model would not work well.
2. Training With Extra Data

I really appreciate your work.

[Jingkang50]

Thank you, @liujunzhuo, for your valuable comments! In OpenOOD, we do have validation set for parameter searching, but seems not use it for OpenGAN. @Prophet-C will take lead to inspect this part this weekend.

[liujunzhuo]

Thanks for your quick reply, I read the code and I think the hyperparam_search() function is used to search for hyperparam, a convenient and easy to use function, which is slightly different from 'open validation' which aims to 'search' for the best GAN-discriminator during training, rather than finding the best hyperparam in a list like finding the best K for KNN. I'm waiting for you inspect, thanks again for your excellent work.

[Jingkang50]

@Prophet-C is still looking into the code. We will update it when finish.

[Prophet-C]

For our current result, we didn't use the model selection method from the paper, this is an oversight which we will fix later. I'll re-run the experiments with the proper model selection method after I fix this. As for the training with extra data, we currently do not consider to include that in the training scheme, as it would be unfair for other methods. We might consider design a Extra Training Data setting which all the methods could utilize later.

[zjysteven]

@Prophet-C I have a fixed version and will push the commits later. You don't have to work on this if you haven't :)

[zjysteven]

@liujunzhuo We've included the "Open Validation" step for OpenGAN here. https://github.com/Jingkang50/OpenOOD/blob/f8870d72e3d0abb0c93c123fed7a2d2800f54c2a/openood/pipelines/train_opengan_pipeline.py#L51-L52 Adding "training with outlier data" for OpenGAN requires significant efforts which we do not consider in the current phase (we do welcome external contributions on this). Note that it is still valid to compare OpenGAN with methods that do not train with outlier data.

[liujunzhuo]

Thank you for your resolving the issue. I'm very appreciate your work!