Vanint
commented
3 years ago Hi, Kaihua,
Thanks for your comments. I think your method in NeurIPS'20 is interesting and insightful, so I also use it as a baseline in my work.
In my view, OOD generalization is very important for DNNs, which is a crucial factor for the deployment of DNNs to real-world applications. From the perspective of concept, OOD generalization is not a specific task. Instead, in my mind, it particularly indicates the situations that training distributions and test distribution are inconsistent, including inconsistent data marginal distributions (e.g., domain adaptation, domain generalization), inconsistent class distributions (e.g., long-tailed recognition, open-set classification), and the combination of the previous two situations (e.g., VQA for sometimes, automatic drive). In comparison, test-agnostic long-tail (LT) in this work is inspired by the conventional LT (typically assuming test class distribution is uniform) and real-world scenarios (test class distributions may not be uniform, e.g. in the automatic drive). Exploring test-agnostic LT can further enhance the practicability of LT methods.
In addition, I think OOD generalization methods may help to handle test-agnostic LT, e.g., better de-biasing the issue of class imbalance. Nevertheless, OOD generalization methods also require further improvement to handle this task, since the test class distributions are unknown and can be any kind of distribution, and simply using OOD generalization methods may be unable to handle different distributions. From the perspective of de-biasing, the long-tailed and inversely long-tailed test class distribution may also be regarded as bias to some degree, but (in my view) it is real scenarios and is not wrong. Meanwhile, in my view, test-agnostic LT may also contribute to the field of OOD generalization, e.g., providing a specific evaluation task by using our provided test-agnostic class distributions (txt) on different LT datasets. In summary, test-agnostic LT (without the prior of test class distributions) not only broadens the research of LT, but is also beneficial to the research of OOD generalization. Therefore, I believe this is an important task and worth further exploring in the future.
BTW, I am also quite interested in causal inference for deep learning, and I think it may be a feasible way to improve the (OOD) generalization of DNNs. Maybe we can add wechat and discuss more.
Best, Yifan
KaihuaTang
Hi,
As a researcher in the long-tailed recognition field, recently, when I started to work on OOD generalization, I found that the long-tailed problem seems to be a special case of OOD generalization, whose definition is quite similar to your test-agnostic LT task. Could you explain their differences? because I'm quite curious about the performance of the OOD algorithms in LT tasks.
Best,