search

Weixin-Liang / MetaShift

MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts (ICLR 2022)
MIT License
105 stars 4 forks source link

distances of the domain generalization experiment #9

Open IbtihalFerwana opened 2 years ago

IbtihalFerwana commented 2 years ago

Hello,

How to reproduce the distances of Table 1 in the paper ? When running the script of dataset/domain_generalization_cat_dog.py, I get 

Distance from dog(cabinet)+dog(bed) to dog(shelf): 0.025

instead of d=0.44

Is there specific data to be included or removed?

IbtihalFerwana commented 2 years ago

Also, can you please let me know how did you specify the size of the training set to be 400 and the testing to be 129 ?

I'm getting very low accuracy for dog(shelf) in the domain generalization experiment

DianeBouchacourt commented 1 year ago

Me too. I get Distance from dog(cabinet)+dog(bed) to dog(shelf): 0.025099006366425383 Distance from dog(bag)+dog(box) to dog(shelf): 0.7551661411955383 Distance from dog(bench)+dog(bike) to dog(shelf): 1.0487592910420251 Distance from dog(boat)+dog(surfboard) to dog(shelf): 1.3317174855037681

DianeBouchacourt commented 1 year ago

In my case, after editing the code quite a bit to finally make main_generalization work on Domain generalization example, I have very strong accuracy ~59% when training on dog(boat)+dog(surfboard). Only the accuracy on dog(shelf) is low (28%) but still not matching at all what is reported in Table 1... I don't know what are the 129 and 400 images though... In my case dog(shelf) is of size 306 images for ex.