search

neulab / ExplainaBoard

Interpretable Evaluation for AI Systems
MIT License
360 stars 36 forks source link

`custom_features` and `custom_analyses` are not deserialized #596

Closed lyuyangh closed 1 year ago

lyuyangh commented 1 year ago

https://github.com/neulab/ExplainaBoard/blob/e111afc94d01cc5ac62e7635bb2122b32b69ac4a/explainaboard/loaders/file_loader.py#L138-L151

data["custom_analyses"] is currently a list[dict]. It should have been deserialized like custom_features because FileLoaderMetadata expects a list[Analysis].

Also, it looks like dataset custom_features & custom_analyses are applied twice. Once with deserialization and the second time without.

https://github.com/neulab/ExplainaBoard/blob/e111afc94d01cc5ac62e7635bb2122b32b69ac4a/explainaboard/loaders/file_loader.py#L647-L666 https://github.com/neulab/ExplainaBoard/blob/e111afc94d01cc5ac62e7635bb2122b32b69ac4a/explainaboard/loaders/file_loader.py#L690-L701