Validate Remaining Top 25 Models as runnable

brain-score / vision

A framework for evaluating models on their alignment to brain and behavioral measurements (50+ benchmarks)

http://brain-score.org

MIT License

126 stars 78 forks source link

Validate Remaining Top 25 Models as runnable #805

Closed pellegreene closed 5 months ago

pellegreene commented 6 months ago

Verify the scores on the remaining Top 25 models:

[x] resnext101_32x16d_wsl
[x] resnet50_finetune_cutmix_e3_robust_linf8255_e0_247x234
[x] effnetb1_cutmix_augmix_sam_e1_5avg_424x377
[x] nasnet_large
[x] fixres_resnext101_32x48d_wsl
[x] antialias-resnet152
[x] custom_model_cv_18_dagger_408
[x] resnet50-VITO-8deg-cc
[x] resnet-50_v2
[x] inception_v4
[x] antialiased-rnext101_32x8d
[x] densenet-201
[x] cv_18_dagger_408_pretrained
[x] inception_v3
[x] AdvProp_efficientnet-b6

pellegreene commented 6 months ago

Notes on two of the models: nasnet_large_pytorch -600G RAM Used slurmstepd: error: Detected 1 oom-kill event(s) in StepId=36556096.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

inception_v4_pytorch File "<__array_function__ internals>", line 6, in concatenate numpy.core._exceptions.MemoryError: Unable to allocate 340. GiB for an array with shape (1000, 91216808) and data type float32

pellegreene commented 6 months ago

Have files for:

resnet50_finetune_cutmix_e3_robust_linf8255_e0_247x234
antialias-resnet152
resnet50-VITO-8deg-cc
resnet-50_v2
antialiased-rnext101_32x8d
AdvProp_efficientnet-b6

Will be validating these and hopefully queueing them up on OM today for a first pass through on MajajHong2015.IT-pls.