Closed qjiang002 closed 1 year ago
Hi @neubig , I got this error when loading this benchmark:
[0] File "/Users/jiangqi/Desktop/Capstone/explainaboard_web/backend/src/gen/explainaboard_web/impl/db_utils/benchmark_db_utils.py", line 187, in <setcomp>
[0] (x.dataset.dataset_name, x.dataset.sub_dataset, x.dataset.split)
[0] AttributeError: 'NoneType' object has no attribute 'dataset_name'
This is because this benchmark try to find all ner systems with 'system_query': {'task_name': 'named-entity-recognition'}
, but there are systems with undefined/custom dataset, so their dataset is None.
One way to deal with undefined datasets is to ignore undefined datasets in benchmark. I think we cannot merge systems with undefined datasets because they may be for different tasks and have different metrics. WDYT?
Hi @neubig , I got this error when loading this benchmark:
[0] File "/Users/jiangqi/Desktop/Capstone/explainaboard_web/backend/src/gen/explainaboard_web/impl/db_utils/benchmark_db_utils.py", line 187, in <setcomp> [0] (x.dataset.dataset_name, x.dataset.sub_dataset, x.dataset.split) [0] AttributeError: 'NoneType' object has no attribute 'dataset_name'
This is because this benchmark try to find all ner systems with
'system_query': {'task_name': 'named-entity-recognition'}
, but there are systems with undefined/custom dataset, so their dataset is None.One way to deal with undefined datasets is to ignore undefined datasets in benchmark. I think we cannot merge systems with undefined datasets because they may be for different tasks and have different metrics. WDYT?
This is another issue not related to this refactor PR. I'll merge this PR and record this problem in another issue.
This PR aims to fix issue #533
Previously,
benchmark_db_utils
needs to loadSystemInfo
which depends on SDK versioning. However, since current benchmark only needsSystem
properties such as overall results for the benchmark plots or tables, we can refactor the code to avoid computing and storing additionalSystemInfo
in cache which will expire due to SDK upgrading.