search

brain-score / core

MIT License
1 stars 6 forks source link

enforce benchmark identifier equivalent to registry identifier #35

Open mschrimpf opened 1 year ago

mschrimpf commented 1 year ago

it is currently possible to construct a case like the following:

class MyBenchmark(BenchmarkBase):
    def __init__(self):
        super(MyBenchmark, self).__init__(
            identifier='identifier2', ...)

benchmark_registry['identifier1'] = MyBenchmark

The identifier in the registry can be different from the identifier in the benchmark itself and we currently have no way of catching it.

To connect to the real world, this has lead to a bug with SyntaxGym in language for example (https://github.com/brain-score/language/pull/183).