[BUG] DeepSpeed on pypi not compatible with latest `numpy`

[younesbelkada]

Describe the bug

Importing deepspeed on a python env with numpy>=2.0.0 fails:

  File "/miniconda3/envs/py39/lib/python3.9/site-packages/deepspeed/autotuning/scheduler.py", line 8, in <module>
    from numpy import BUFSIZE
E   cannot import name 'BUFSIZE' from 'numpy' (/miniconda3/envs/py39/lib/python3.9/site-packages/numpy/__init__.py)

To Reproduce

pip install deepspeed on a env with python>=3.9 and import deepspeed

[loadams]

Thanks @younesbelkada, I believe the second linked PR should resolve this.

[adk9]

@loadams, is it really necessary for us to pin numpy to a pre-2.0.0 version?

I wonder if BUFSIZE was a botched import and the right fix for this is something like:

diff --git a/deepspeed/autotuning/scheduler.py b/deepspeed/autotuning/scheduler.py
index 7d2a1c08..35f709ea 100755
--- a/deepspeed/autotuning/scheduler.py
+++ b/deepspeed/autotuning/scheduler.py
@@ -5,7 +5,6 @@

 import copy

-from numpy import BUFSIZE
 import json
 import subprocess
 import sys
@@ -18,7 +17,7 @@ import hjson
 from tqdm import tqdm

 from ..utils import logger
-from .constants import AUTOTUNING, AUTOTUNING_METRIC_PATH
+from .constants import AUTOTUNING, AUTOTUNING_METRIC_PATH, BUFSIZE
 from .utils import get_val_by_key, search_error, was_interruptted
 """
 thread-0: loop over experiment queue dispatching experiments if they become available

Um, lmk if this makes sense and I can open a PR for it.

[loadams]

Okay @younesbelkada, DeepSpeed master branch should have this resolved here, one we confirm, we can do a new release as well.

https://github.com/microsoft/DeepSpeed/pull/5680

[younesbelkada]

Thank you very much for the quick action @loadams !

[sedletsky-f5]

Okay @younesbelkada, DeepSpeed master branch should have this resolved here, one we confirm, we can do a new release as well.

5680

please release the fix, to unblock users