Some questions about reported metrics

[sweetdream779]

Hello, thank you for your work!

I have some clarification questions about your reported metrics:

• In case of out-of-distribution evaluations, you trained several models with different subsets and reported "their weighted average performance." But which weights did you use or how can I calculate them?
• In case of Wilder Testsets evaluation, did you use your general classifier which were trained on Arbitrary-domains & Arbitrary-models data(Testbed 4)?

Thanks!

[yafuly]

Hi,

• "weighted average performance" means weighting the performance by the size of the testset.
• Yes, the classifier was trained on "Arbitrary-domains & Arbitrary-models data", with the decision boundary adjusted.

Thank you for your interest in our work. Please feel free to reach out with any additional questions.

[sweetdream779]

Thanks for the answer! One more clarification question about weighting. Am I calculating the weights correctly?

sizes = np.array([[7420, 5386, 9320, 1838, 13284, 2884, 16024]])
weights = sizes / np.sum(sizes)

[yafuly]

Yes, your calculation of the weights is correct.