search

brain-score / vision

A framework for evaluating models on their alignment to brain and behavioral measurements (50+ benchmarks)
http://brain-score.org
MIT License
120 stars 74 forks source link

Disallow nonconverted models to be scored #1082

Open mike-ferguson opened 1 month ago

mike-ferguson commented 1 month ago

In scoring, right now if you score on "all models", Jenkins will try to score models that have been nonconverted (1.0 models).

mschrimpf commented 1 month ago

I think we should not make this specific to the 2.0 conversion (which should resolve itself once all models have been converted), but rather how to deal with models that aren't working anymore (for instance because dependencies are incompatible, weights no longer accessible, etc)