search

facebookresearch / DomainBed

DomainBed is a suite to test domain generalization algorithms
MIT License
1.37k stars 294 forks source link

A question about in/out accuracy #141

Closed noparkee closed 4 months ago

noparkee commented 1 year ago

Hello. I have a question about in/out accuracy when measuing performance.

If there are n domains for a dataset, we divide each domain into a train set (in) and a test set (out). Then, train the model using the train data of the domains except the target domain. After model selection, when measuring performance, why do we evaluate with the test_in_acc_key of the target domain, i.e., the train set? Why not the whole dataset and why not the test set?

Thank you!

abalhomaid commented 9 months ago

Hello, I think there is a bit of confusion here.

This is a good thread to a similar question explaining what in/out accuracy means https://github.com/facebookresearch/DomainBed/issues/53#issuecomment-849509189

If I understood correctly, env_x_out is a smaller split used to identify hyperparameters and env_x_in is for training and final evaluation. Hope this helps.

piotr-teterwak commented 4 months ago

Yes, @abalhomaid is correct. This is a design decision, and including the full dataset could make checkpoint selection difficult for some algorithms (you need some validation data for this). Closing. Feel free to re-open if there is still confusion.