Getting all the ground-truth labels for each eval-setting?

[vishaal27]

Hey, is there a way to query the db to get the ground-truth sample-level labels per eval-setting, is this stored in your db somewhere? Or will I have to manually eval all the settings once to do this? Essentially, it would be super helpful for me to know the quickest way to get them.

[rtaori]

Yeah they're not stored in the DB directly (since for many datasets the labels are based on the file structure), so you'd need to run through all of them once (perhaps with a dummy model) to get them out

[vishaal27]

Cool thanks, that's helpful to know!

[vishaal27]

Hey @rtaori, sorry for reopening the issue with another question and my repeated barrage of questions, but this should hopefully be one of the last ones: Are the sample-wise ground-truth labels consistent across all the pgd datasets? Specifically, I am interested in these 4 datasets: pgd.l2.eps0.5, pgd.linf.eps0.5, pgd.linf.eps2 and pgd.l2.eps0.1. However, while running the eval scripts to save the targets, I run into OOM issues for the pgd.l2 variants. Since the pgd.linf variants have exactly the same targets, I assume this would also be the case with the pgd.l2 ones? I guess you run pgd on the imagenet val set directly without perturbing the input sample ordering so this is what should be happening anyway, but I just wanted to confirm that I did not miss something, thanks!

[rtaori]

Haha no worries. Yes the targets are exactly the same for all pgd datasets, as they're all based on the imagenet val set as you said.

[vishaal27]

Awesome thanks!