When a framework is not available in frameworks_X.yaml but is available in frameworks.yaml and you invoke python runbenchmark.py framework:X the error message will still read:
ERROR:
Incorrect framework `X`: not listed in ['/Users/pietergijsbers/repositories/automlbenchmark/resources/frameworks.yaml'].
The error message should refer to frameworks_X.yaml as that is actually the one checked against.
In general it looks like the definition loading can use some refactoring (e.g., maybe we want to allow loading only specific files based on the tag).
When a framework is not available in
frameworks_X.yamlbut is available inframeworks.yamland you invokepython runbenchmark.py framework:Xthe error message will still read:The error message should refer to
frameworks_X.yamlas that is actually the one checked against.In general it looks like the definition loading can use some refactoring (e.g., maybe we want to allow loading only specific files based on the tag).