nclskfm / square-core

SQuARE: Software for question answering research.
https://square.ukp-lab.de
2 stars 0 forks source link

21 support for different metrics #31

Closed BenediktStuhrmann closed 1 year ago

BenediktStuhrmann commented 1 year ago

What does this PR do?

Added functionality for mapping datasets to a universal/generic representation (per skill-type). Currently supports extractive-qa and multiple-choice-qa. Added functionality for mapping predictions to metric-format. Added functionality for mapping references (in universal dataset-format) to metric-format. Integrated mapping of dataset and mapping for metrics into predict and evaluate endpoints.

For testing: Extractive-QA Skill: 62c1ae1b536b1bb18ff91ce3 Mocked dataset-metadatas: squadand quoref

Multiple-Choice Skill: 62c1ae19536b1bb18ff91cde Mocked dataset-metadata: commonsense_qa and cosmos_qa

Currently supported metrics: exact_match, squad, squad_v2

Before submitting / marking as 'ready to review'

nclskfm commented 1 year ago

Hast noch 1 merge conflict @BenediktStuhrmann

BenediktStuhrmann commented 1 year ago

Hast noch 1 merge conflict @BenediktStuhrmann

Fixed