Convert benchmark competition models

[deirdre-k]

Once all of the "important" models have been tracked down, they may not be in converted form. Martin has provided the model conversion script that has been used previously. If that script works, then all we would need to do is create a map of these models(i.e. model_x converted into model_y) in an Airtable as described in this board. Once this is completed, we can move onto the model score validation against the website scores.

Model conversions tracked here. Benchmark competition models here

• [x] cvt_cvt-w24-384-in22k_finetuned-in1k_4
• [x] resnext101_32x8d_wsl
• [x] effnetb1_cutmixpatchSAM
• [x] effnetb1_cutmixpatch_augmix_robust32_avge4e7_manylayers_324x288 (Winner of the 2022 competition)
• [x] resnext101_32x32d_wsl
• [x] effnetb1_272x240
• [x] resnext101_32x48d_wsl
• [x] pnasnet_large
• [x] resnet-152_v2
• [x] focalnet_tiny_lrf_in1k
• [x] hmax
• [x] alexnet
• [x] CORnet-S
• [x] resnet-50-robust
• [x] voneresnet-50-non_stochastic
• [x] resnet18-local_aggregation
• [x] grcnn_robust_v1 (Top-3 competition 2022)
• [x] custom_model_cv_18_dagger_408 (Top-3 competition 2022)
• [x] ViT_L_32_imagenet1k
• [x] mobilenet_v2_1.4_224
• [x] pixels

Acceptance Criteria: Have converted models that can be run in BrainScore 2.0.

[deirdre-k]

Note: Tensorflow models are outdated and not working well, converting those to PyTorch

[pellegreene]

All benchmark models have been converted.

Some notes:

• grcnn_robust_v1 is the model where we have contacted the author for model weights
• voneresnet-50-non_stochastic and CORnet-S are throwing errors from some dicarlolab imports, working to fix those now(but they now both pull activations!)
• pnasnet_large and resent-152_v2 are both running for scores.

[pellegreene]

Also, currently locating weights on s3 for custom_model_cv_18_dagger_408.

[pellegreene]

Of the 21 models in the benchmark competition, here is the breakdown:

• 8 reproduce
• 11 do not reproduce
• Of the 11 that do not reproduce, 7 improve upon the website score score.
• 2 are not runnable
• grcnn needs weights from the author
• CORnet-S is throwing an error on the PCA step

[pellegreene]

We had run a small handful of models on the incorrect benchmark(MajajHong2015public.IT-pls instead of MajajHong2015.IT-pls), here are the models that were rerun and some notes on them:

• resnet-50-robust now lines up with the website score.
• pnasnet_large generates a score of .521 vs the website's .526. This was a model converted from TensorFlow, so some variance is to be expected. In this instance, it is at least slight.
• resnet-152_v2 produced a score of 0.08304313 vs the website's score of .532. This was also a TF model, but this is an enormous outlier. The conversion is just making use of the torchvision weights. I'll try running on some other region benchmarks and see what comes out of it.
• Rerunning mobilenet_v2_1-4_224 on MajajHong2015.IT-pls results in the following error: line 71, in get_layer assert module is not None, f"No submodule found for layer {layer_name}, at part {part}" AssertionError: No submodule found for layer features.0.2, at part features

I've forwarded these notes to Martin, waiting to hear back on what he would advise I do.

[pellegreene]

Figured out the previous Mobilenet error, it was a results caching issue. It is now throwing the following error when I run it: File "/om/weka/quest/epellegr/projects/model_validation/vision/brainscore_vision/model_helpers/activations/core.py", line 110, in _from_paths_stored return self._from_paths(layers=layers, stimuli_paths=stimuli_paths, require_variance=require_variance) File "/om/weka/quest/epellegr/projects/model_validation/vision/brainscore_vision/model_helpers/activations/core.py", line 119, in _from_paths return self._package(layer_activations=layer_activations, stimuli_paths=stimuli_paths, require_variance=require_variance) File "/om/weka/quest/epellegr/projects/model_validation/vision/brainscore_vision/model_helpers/activations/core.py", line 245, in _package axis=layer_assemblies[0].dims.index('neuroid')) IndexError: list index out of range

[pellegreene]

Moving this to Done as all models have been converted.