deepspeed incompatibility?

[pranavbhatkhande]

Just cloned the repo and followed instructions

  error: subprocess-exited-with-error

  × Running setup.py install for deepspeed did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      [2024-06-09 12:42:25,748] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
      [2024-06-09 12:42:26,334] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-f5hoq4i3/deepspeed_246c7af21ae0437ba36691236b7bc310/setup.py", line 100, in <module>
          cuda_major_ver, cuda_minor_ver = installed_cuda_version()
        File "/tmp/pip-install-f5hoq4i3/deepspeed_246c7af21ae0437ba36691236b7bc310/op_builder/builder.py", line 47, in installed_cuda_version
          import torch.utils.cpp_extension
        File "/home/pranav/sandboxes/speech-dataset-generator/venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 28, in <module>
          from pkg_resources import packaging  # type: ignore[attr-defined]
      ImportError: cannot import name 'packaging' from 'pkg_resources' (/home/pranav/sandboxes/speech-dataset-generator/venv/lib/python3.10/site-packages/pkg_resources/__init__.py)
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> deepspeed

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[davidmartinrius]

As explained in https://github.com/davidmartinrius/speech-dataset-generator?tab=readme-ov-file#installation it has not been tested on Windows. I guess you tried to install it on Windows or another os different than Linux?

In any case, pull requests are welcome if you want to fix it.

Thanks

[pranavbhatkhande]

@davidmartinrius I'm on PopOS, which is basically Ubuntu.. looking into it. Cheers

[davidmartinrius]

Ok... I understand.. well if you are using this OS I will install a fresh install in my Ubuntu to try to reproduce this error. As this project depends on other projects perhaps there was an update to one of them an deepspeed became incompatible. But it is an assumption. When I have tried it I'll let you know.

Thanks

[pranavbhatkhande]

Its definitely a dependency problem, seems to come from git+https://github.com/davidmartinrius/resemble-enhance.git, I will try switching this out to the main branch. Did you add any special commits to your branch on resemble-enhance to make this project work? Happy to help, I will take a look this week as well!

[davidmartinrius]

I fixed it. It was crashing because the latest version of setuptools. I updated the requirements.txt, please pull the changes try to reinstall and, please confirm if it worked for you.

[pranavbhatkhande]

@davidmartinrius I just tried it, on your latest commit, but maybe this is a separate issue now?

  error: subprocess-exited-with-error

  × Running setup.py install for deepspeed did not run successfully.
  │ exit code: 1
  ╰─> [11 lines of output]
      [2024-06-10 15:27:49,739] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
      [2024-06-10 15:27:50,326] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
      DS_BUILD_OPS=0
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-a_n3fyxw/deepspeed_3de91b2ad25944768c7f136a4542ac50/setup.py", line 173, in <module>
          op_compatible = builder.is_compatible()
        File "/tmp/pip-install-a_n3fyxw/deepspeed_3de91b2ad25944768c7f136a4542ac50/op_builder/evoformer_attn.py", line 53, in is_compatible
          with open(f'{self.cutlass_path}/CHANGELOG.md', 'r') as f:
      FileNotFoundError: [Errno 2] No such file or directory: '/home/pranav/sandboxes/speech-dataset-generator/cutlass/CHANGELOG.md'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> deepspeed

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
(venv) pranav:~/sandboxes/speech-dataset-generator (main)$ git log -1 | cat 
commit 5142e9779dc06f0ad1540bc151979d92a5627739
Author: David Martin Rius <0991592@gmail.com>
Date:   Mon Jun 10 20:52:50 2024 +0200

    Update requirements.txt

    add specific version of setuptools. The latest one make deepspeed crash

[davidmartinrius]

Have you created a new environment? Or you reused the old one? After your last message I cloned and reinstalled the project from scratch and it worked well...

[davidmartinrius]

Closed due to inactivity

[pranavbhatkhande]

@davidmartinrius sorry I didn't get back to you. My cutlass path was the issue, your fix works. Cheers!