search

AnswerDotAI / bert24

Apache License 2.0
25 stars 3 forks source link

Add EurLex to evaluation suite #65

Closed rbiswasfc closed 2 weeks ago

rbiswasfc commented 2 weeks ago

This PR adds EurLex (multi-label classification task focusing on legal domain) to the evaluation suite. EurLex dataset has several variations -- here, the one from LexGLUE is used to have a good reference score to match.

EurLex job can be run with python glue.py yamls/baselines/deberta-v3-long-context.yaml. With the current config, it achieves 72.30 micro-f1 score with debeta-v3-large model (LexGLUE reference: 72.1).

rbiswasfc commented 2 weeks ago

While running the tests, the following two tests failed:

FAILED tests/test_glue.py::test_glue_script[mosaic_bert] - AttributeError: 'BertConfig' object has no attribute 'normalization'
FAILED tests/test_superglue.py::test_superglue_script[mosaic_bert] - AttributeError: 'BertConfig' object has no attribute 'normalization'

I'm not sure if these are related to current PR -- as it doesn't modify mosaic_bert any specific code.

warner-benjamin commented 2 weeks ago

@rbiswasfc I just ran the tests on main and they all passed. Maybe this branch is too out of date?

rbiswasfc commented 2 weeks ago

@rbiswasfc I just ran the tests on main and they all passed. Maybe this branch is too out of date?

Thanks for running the tests, helped with debugging! I found the cause and pushed a fix. All tests are fine now.

I wanted to pass problem_type= multi_label_classification to hf_bert for EurLex and hence originally added this line main_config.model.model_config = task_config.get("model_config", {}). However, it reset the model config for mosaic_bert. Fixed this with: main_config.model.model_config.update(task_config.get("model_config", {})).