Issue811: Implement holdout estimation

deepset-ai / FARM

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

https://farm.deepset.ai

Apache License 2.0

1.74k stars 247 forks source link

Issue811: Implement holdout estimation #818

Closed johann-petrak closed 3 years ago

johann-petrak commented 3 years ago

Datasilo and example program for holdout estimation. This also does stratification of the dev split.

Timoeller commented 3 years ago

Thanks for the addition. This addition already looks quite good. Do you think it would make sense to add tests to the newly created DataSiloForHoldout?

johann-petrak commented 3 years ago

I would prefer to close this PR and provide a new clean PR that solves issues 817, 818 and 819 together. See my comment in https://github.com/deepset-ai/FARM/pull/817 and issue https://github.com/deepset-ai/FARM/issues/819

johann-petrak commented 3 years ago

Closing, I will provide a new PR for this (see https://github.com/deepset-ai/FARM/issues/819#issuecomment-874194965) Sorry for the confusion.