Update datasets requirement from <2.20,>=2.18 to >=2.18,<2.22

Updates the requirements on datasets to permit the latest version.

Release notes

2.21.0

Features
Support pyarrow large_list by @albertvillanova in huggingface/datasets#7019
Support Polars round trip:
import polars as pl
from datasets import Dataset
df1 = pl.from_dict({"col_1": [[1, 2], [3, 4]]}
df2 = Dataset.from_polars(df).to_polars()
assert df1.equals(df2)
What's Changed

Use HF_HUB_OFFLINE instead of HF_DATASETS_OFFLINE by @Wauplin in huggingface/datasets#6968

packaging: Remove useless dependencies by @daskol in huggingface/datasets#6971

Fix resuming arrow format by @lhoestq in huggingface/datasets#6964

Fix webdataset pickling by @lhoestq in huggingface/datasets#6972

Set temporary numpy upper version < 2.0.0 to fix CI by @albertvillanova in huggingface/datasets#6975

Fix regression for pandas < 2.0.0 in JSON loader by @albertvillanova in huggingface/datasets#6978

Ensure compatibility with numpy 2.0.0 by @KennethEnevoldsen in huggingface/datasets#6976

Remove underlines between badges by @novialriptide in huggingface/datasets#6966

Update docs on trust_remote_code defaults to False by @albertvillanova in huggingface/datasets#6981

Improve skip take shuffling and distributed by @lhoestq in huggingface/datasets#6965

Fix tests using hf-internal-testing/librispeech_asr_dummy by @albertvillanova in huggingface/datasets#6998

Fix dump of bfloat16 torch tensor by @lhoestq in huggingface/datasets#7002

minor fix for bfloat16 by @lhoestq in huggingface/datasets#7003

Fix incorrect rank value in data splitting by @yzhangcs in huggingface/datasets#6994

less script docs by @lhoestq in huggingface/datasets#6993

Fix CI by temporarily pinning ruff < 0.5.0 by @albertvillanova in huggingface/datasets#7007

Support ruff 0.5.0 in CI by @albertvillanova in huggingface/datasets#7009

Fix WebDatasets KeyError for user-defined Features when a field is missing in an example by @ProGamerGov in huggingface/datasets#7004

[Streaming] retry on requests errors by @lhoestq in huggingface/datasets#6963

Re-enable raising error from huggingface-hub FutureWarning in CI by @albertvillanova in huggingface/datasets#7011

Skip faiss tests on Windows to avoid running CI for 360 minutes by @albertvillanova in huggingface/datasets#7014

Support fsspec 2024.6.1 by @albertvillanova in huggingface/datasets#7017

Persist IterableDataset epoch in workers by @lhoestq in huggingface/datasets#6710

Fix casting list array to fixed size list by @albertvillanova in huggingface/datasets#7021

Remove dead code for pyarrow < 15.0.0 by @albertvillanova in huggingface/datasets#7023

Fix check_library_imports by @lhoestq in huggingface/datasets#7026

Missing line from previous pr by @lhoestq in huggingface/datasets#7027

Fix ci by @lhoestq in huggingface/datasets#7028

Add decorator as explicit test dependency by @albertvillanova in huggingface/datasets#7043

Mark tests that require librosa by @albertvillanova in huggingface/datasets#7044

Unblock NumPy 2.0 by @NeilGirdhar in huggingface/datasets#6991

Fix tensorflow min version depending on Python version by @albertvillanova in huggingface/datasets#7045

Support librosa and numpy 2.0 for Python 3.10 by @albertvillanova in huggingface/datasets#7046

add checkpoint and resume title in docs by @lhoestq in huggingface/datasets#7050

Update load_hub.mdx by @severo in huggingface/datasets#7057

Add batching to IterableDataset by @lappemic in huggingface/datasets#7054

... (truncated)

Commits

a1b5a32 Release: 2.21.0 (#7098)
83e5c05 Fix CI for metrics
af818af chore: fix typos in docs (#7034)
be43e09 Support pyarrow large_list (#7019)
e7094be Support HTTP authentication in non-streaming mode (#7082)
5735e6b feat: support non streamable arrow file binary format (#7025)
4d5e237 Fix push_to_hub by not calling create_branch if branch exists (#7069)
87e8ab3 Set load_from_disk path type as PathLike (#7081)
360d8c1 Fix prepare_single_hop_path_and_storage_options (#7068)
779f6d7 Fix CI test_convert_to_parquet (#7078)
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

LoicGrobol / zeldarose

Update datasets requirement from <2.20,>=2.18 to >=2.18,<2.22 #104

2.21.0

Features

What's Changed