search

huggingface / huggingface_hub

The official Python client for the Huggingface Hub.
https://huggingface.co/docs/huggingface_hub
Apache License 2.0
2.02k stars 531 forks source link

Torch: test on 2.0 and latest versions + explicitly load with `weights_only=True` #2488

Closed Wauplin closed 1 month ago

Wauplin commented 1 month ago

To merge before https://github.com/huggingface/huggingface_hub/pull/2440 (needed to test it). Two things in this PR:

  1. we are now running the torch-test CI with torch~=2.0 and torch=="latest". This is to ensure a sense of backward-compatibility and still testing on latest updates. 2.0 was chosen arbitrarily. It has been released ~15 months ago. It doesn't mean huggingface_hub does not support torch 1.0 (we would have received reports^^), just that we are not guaranteeing it.

  2. Pytorch tests are currently failing in CI with this FutureWarning:

_FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature._

This is actually a good opportunity to add weights_only=True explicitly in the Pytorch hub mixin when loading from an unsafe pickle. Note that the default file type is still .safetensors when saving / loading weights.

Expectations: a green CI and relaxed users!

HuggingFaceDocBuilderDev commented 1 month ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Wauplin commented 1 month ago

It's not a hard requirements on our side no. I just needed to chose one but happy to align with transformers on torch==1.11. I've pushed https://github.com/huggingface/huggingface_hub/pull/2488/commits/a3a3b4571a54c71444b1d7360c942b975bebdb82 in that sense. I'll merge once CI is green.

Wauplin commented 1 month ago

Tests are passing on torch 1.11 so all good to test this one in the CI!