Open albertz opened 1 year ago
I guess the most canonical way would be to write a RETURNN Dataset for this. Maybe derived from CachedDataset2.
I already implemented this as a custom dataset some time ago (see implementation below).
Things to discuss are maybe how to handle tokenisation and other preprocessing steps.
In the current implementation there is either the option to define a map function (see https://huggingface.co/docs/datasets/about_map_batch) in the config file or to provide a preprocessed dataset stored with save_to_disk
.
num_outputs and data_type is currently extracted from the dataset features attribute (which may not always be available).
Another point to discuss is how to handle caching for large datasets (currently the default caching mechanism of hf datasets is used which does not fit very well with our setups).
Related is the Sisyphus job to prepare HuggingFace datasets (https://github.com/rwth-i6/i6_core/pull/253). Doesn't this handle the caching? Ideally we should prepare our dataset wrapper here such that it works properly together with this download preparation job.
@dthulke Can you say some examples what HF datasets you use?
Related is the Sisyphus job to prepare HuggingFace datasets (https://github.com/rwth-i6/i6_core/pull/253). Doesn't this handle the caching? Ideally we should prepare our dataset wrapper here such that it works properly together with this download preparation job.
Yes, this handles the caching of the initial dataset download, but not the caching of the processed version (via dataset.map
). But we could add a separate job for this.
@dthulke Can you say some examples what HF datasets you use?
In RETURNN, I mainly use hf datasets for sequence classification (e.g. sentiment analysis) or sequence tagging task (named entity recognition). For example: https://huggingface.co/datasets/conll2003
In addition, I have a few datasets the I load with custom dataset loading scripts or the default json dataset implementation.
One example, for additional preprocessing (beyond tokenisation) is to include document-level/cross-sentence context or to convert NER labels (given as start end positions) to BIO labels.
We should be able to use HuggingFace datasets directly in RETURNN.
I guess the most canonical way would be to write a RETURNN
Dataset
for this. Maybe derived fromCachedDataset2
.A separate independent more direct PyTorch dataset wrapper might make sense. Or actually I think not needed, as HuggingFace already directly supports this?