This PR tweaks the path to download the references from the Hugging Face Hub, so that each file is stored in a separate folder, e.g.:
~/dataset_name/dataset_name.json
This is needed because dataset repositories have a limit of 2k files per folder, and we exceeded this limit on a recent extraction of references I performed.
This PR tweaks the path to download the references from the Hugging Face Hub, so that each file is stored in a separate folder, e.g.:
This is needed because dataset repositories have a limit of 2k files per folder, and we exceeded this limit on a recent extraction of references I performed.
cc @sebastianGehrmann