checkpoint and resume an iterable dataset (e.g. when streaming):
>>> iterable_dataset = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)
>>> for idx, example in enumerate(iterable_dataset):
... print(example)
... if idx == 2:
... state_dict = iterable_dataset.state_dict()
... print("checkpoint")
... break
>>> iterable_dataset.load_state_dict(state_dict)
>>> print(f"restart from checkpoint")
>>> for example in iterable_dataset:
... print(example)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore ` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore ` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions
Bumps the dependencies group with 4 updates: pydantic, filelock, ruff and datasets.
Updates
pydantic
from 2.7.3 to 2.7.4Release notes
Sourced from pydantic's releases.
Changelog
Sourced from pydantic's changelog.
Commits
1ab655a
Prep for v2.7.4 release (#9642)2e3650e
V1 bump to v1.10.16 (#9639)Updates
filelock
from 3.14.0 to 3.15.1Release notes
Sourced from filelock's releases.
Commits
3d7b1a6
Hotfix: Restore init method; more robust initialization for singleton loc...c64787f
Don't initialize BaseFileLock when just returning existing instance (#334)87453f3
asyncio support (#332)0ee2e3c
[pre-commit.ci] pre-commit autoupdate (#333)8292363
[pre-commit.ci] pre-commit autoupdate (#330)1cdb3b4
[pre-commit.ci] pre-commit autoupdate (#328)19f78fc
[pre-commit.ci] pre-commit autoupdate (#327)8a168c5
[pre-commit.ci] pre-commit autoupdate (#326)Updates
ruff
from 0.4.8 to 0.4.9Release notes
Sourced from ruff's releases.
Changelog
Sourced from ruff's changelog.
Commits
4f49e91
Bump version to v0.4.9 (#11872)d681a45
Makeruff_db
a required crate forruff_python_semantic
(#11874)89bb07c
UPDATE latest supported versions to 3.13 (#11870)fe462b3
Update Python compatibility to 3.13 (#11861)c5bc368
[red-knot] ImproveVfs
andFileSystem
documentation (#11856)73370fe
Usestarts_with('/')
instead ofis_absolute
to avoid platform specific AP...22b6488
red-knot: Add directory support toMemoryFileSystem
(#11825)d4dd96d
red-knot:source_text
,line_index
, andparsed_module
queries (#11822)efbf7b1
red-knot[salsa part 2]: Setup semantic DB and Jar (#11837)9dc226b
Add supported commands in server capabilities (#11850)Updates
datasets
from 2.19.2 to 2.20.0Release notes
Sourced from datasets's releases.
... (truncated)
Commits
98fdc9e
Release: 2.20.0 (#6969)af3acfd
fix(ci): remove unnecessary permissions (#6962)37a6036
Move info_utils errors to exceptions module (#6952)9510252
Better error handling indataset_module_factory
(#6959)97513be
feat(ci): add trufflehog secrets detection (#6960)686f5df
Add support for categorical/dictionary types (#6892)a2dc287
Remove defaulttrust_remote_code=True
(#6954)09ebf51
Validate config name and data_files in packaged modules (#6915)5bbbf1b
Validate config name and data_files in packaged modules (#6915)6548e0e
Fix typos in docs (#6957)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase
.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show