Closed johann-petrak closed 3 years ago
Thanks for combining the contributions into one very dense PR.
I think the addition of the ConcatTensorDataset might have broken something for other tasks, see the failing CI. Could you look into it? Maybe separate the additional ConcatTensorDataset from other tasks as a workaround?
Disclaimer: I will only be available for reviewing further adjustments from Monday onwards again.
Sorry this was really caused by a stupid typo I introduced when I tried to avoid a warning message being too long in a single line.
The ConcatTensorDataset has been implemented as a subclass of ConcatDataset and literally works exactly the same unless one requests a slice or list of indices instead of a single index - which can never happen in any code not added by me here, because that would have been impossible so far.
I did run the tests here of which one fails: test_ner_amp.py::test_ner_amp but I suspect due to completely different/unrelated reasons. The tests in github actions seem to pass now: https://github.com/deepset-ai/FARM/runs/3013430217?check_suite_focus=true
btw as slight improvement for the next PR please mention "fixes issue num" explicitly in the PR description, then the corresponding issues get closed automatically.
This should fix the following issues:
The following notes are important:
dev_stratification
and it has a non-None value. Otherwise, the behaviour is exactly as before, for backwards compatibility and also to make sure that this does not impact any other situations with different tasks, multihead etc.dev_stratification
is specified and not None: if it is False, no stratification is done, but the split is still done properly independently of the chunk sizestratified
parameter of the DatSiloForCrossVal make factory function tostratification
to match the parameter namedev_stratification
in TextClassificationProcesser, where that name was chosen because the parameter is not just boolean but can also be None (for the old way of doing things) or a callable for one's own way to perform the split.I have not tested this with more complex multihead models - yet: i will use this myself with multihead models, so should there be issues with that I will correct them then.