huggingface / accelerate

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support
https://huggingface.co/docs/accelerate
Apache License 2.0
7.97k stars 970 forks source link

Use `numpy._core` instead of `numpy.core` #3247

Closed qgallouedec closed 1 day ago

qgallouedec commented 2 days ago

What does this PR do?

We've been getting this warning:

/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.11/site-packages/accelerate/utils/other.py:220: DeprecationWarning: numpy.core is deprecated and has been renamed to numpy._core. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.multiarray. np.core.multiarray._reconstruct,

Before submitting

Who can review?

@muellerzr @BenjaminBossan @SunMarc

BenjaminBossan commented 2 days ago

The issue here is that np._core does not exist in older numpy versions (probably < 2.0 but I haven't checked), so this would fail there (as happens in the failing doc build). We would need to do a version check and use either np.core or np._core.

qgallouedec commented 2 days ago

good point!

The very first version that includes np._core is ~1.26.1~ 2.0.0.

What about something like

np_core = np._core if version.parse(np.__version__) >= version.parse("2.0.0") else np.core

alternatively we can do something like

np_core = getattr(np, "_core", np.core)

but I find it less intuitive to know when the condition is met or not, and we'll probably forget to remove this if/else once we've increased the min version of numpy

EDIT: min version is actually 2.0.0

qgallouedec commented 2 days ago

wait it seems like np._core for np<2.0 is not the same as for np>=2.0. Checking right now

EDIT: version threshold is actually 2.0.0 (not 1.26.1). Editing my above comment

HuggingFaceDocBuilderDev commented 2 days ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.