neulab / ExplainaBoard

Interpretable Evaluation for AI Systems
MIT License
360 stars 36 forks source link

Add system metadata class #575

Open odashi opened 1 year ago

odashi commented 1 year ago

Processor.process() takes metadata, which is used to directly initialize SysOutputInfo. However, these are essentially different data (especially, "metadata" $\subset$ SysOutputInfo, but not $=$) and the current implementation makes some confusion around this:

The most significant abuse around this behavior is that FileLoaderMetadata is implicitly converted into SysOutputInfo. This shouldn't work unless explicit conversion: https://github.com/neulab/ExplainaBoard/blob/4cec0a01cbe2617e9a67a440be25ee4252f792b2/integration_tests/ner_test.py#L148-L154

To this end, we need:

odashi commented 1 year ago

RFC: @neubig @pfliu-nlp

neubig commented 1 year ago

Sounds like a good idea, I agree

pfliu-nlp commented 1 year ago

"The most significant abuse around this behavior is that FileLoaderMetadata is implicitly converted into SysOutputInfo."

Agree. Adding system metadata class sounds nice.