Added functionality for mapping datasets to a universal/generic representation (per skill-type). Currently supports extractive-qa and multiple-choice-qa.
Added functionality for mapping predictions to metric-format.
Added functionality for mapping references (in universal dataset-format) to metric-format.
Integrated mapping of dataset and mapping for metrics into predict and evaluate endpoints.
For testing:
Extractive-QA Skill: 62c1ae1b536b1bb18ff91ce3
Mocked dataset-metadatas: squadand quoref
Multiple-Choice Skill: 62c1ae19536b1bb18ff91cde
Mocked dataset-metadata: commonsense_qa and cosmos_qa
Currently supported metrics: exact_match, squad, squad_v2
Before submitting / marking as 'ready to review'
[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
What does this PR do?
Added functionality for mapping datasets to a universal/generic representation (per skill-type). Currently supports extractive-qa and multiple-choice-qa. Added functionality for mapping predictions to metric-format. Added functionality for mapping references (in universal dataset-format) to metric-format. Integrated mapping of dataset and mapping for metrics into predict and evaluate endpoints.
For testing: Extractive-QA Skill:
62c1ae1b536b1bb18ff91ce3
Mocked dataset-metadatas:squad
andquoref
Multiple-Choice Skill:
62c1ae19536b1bb18ff91cde
Mocked dataset-metadata:commonsense_qa
andcosmos_qa
Currently supported metrics:
exact_match
,squad
,squad_v2
Before submitting / marking as 'ready to review'