search

WangYueFt / rfs

366 stars 66 forks source link

A potential issue that might affect comparison fairness #21

Open aiyolo opened 3 years ago

aiyolo commented 3 years ago

https://github.com/WangYueFt/rfs/blob/f8c837ba93c62dd0ac68a2f4019c619aa86b8421/eval_fewshot.py#L126-L131 In experiment, the flag of num_workers=3 in meta_testloader, it will assign 3 threads to fetch data, however, these threads have the same seed for np.random, which means it will sample 3 similar task, the 3 task has the same class, the same support images and the same query images, so it might be unfair to other methods although it is irrelevant to the performance. By the way, setting worker_it_fn for each thread or num_workers<=1 might be a remedy.

Jiawen-huang commented 3 years ago

I solve it. Just delete this sentence: lambda x: np.asarray(x),

aiyolo commented 3 years ago

I don't think so, the line you commented maps x to a np.Array, which has nothing to with a random seed.