"packaging" library exists but not found

[mahmoodn]

I use the following command to build Apex, but I don't know why it doesn't find packaging library.

$ pip3 install --user -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Using pip 22.1.1 from /home/mahmood/.local/lib/python3.8/site-packages/pip (python 3.8)
WARNING: Ignoring invalid distribution -umpy (/home/mahmood/.local/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -umpy (/home/mahmood/.local/lib/python3.8/site-packages)
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /disk1/mahmood/training/large_language_model/megatron-lm/apex
  Running command pip subprocess to install build dependencies
  WARNING: Ignoring invalid distribution -umpy (/home/mahmood/.local/lib/python3.8/site-packages)
  Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com, https://pypi.ngc.nvidia.com
  Collecting setuptools
    Downloading setuptools-70.0.0.tar.gz (2.3 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 28.2 MB/s eta 0:00:00
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting wheel
    Downloading wheel-0.43.0.tar.gz (99 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 252.2 MB/s eta 0:00:00
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: wheel, setuptools
    Building wheel for wheel (pyproject.toml): started
    Building wheel for wheel (pyproject.toml): finished with status 'done'
    Created wheel for wheel: filename=wheel-0.43.0-py3-none-any.whl size=65775 sha256=b1d9badbcda1857993ce58fc40356a18f7cb87e513a06b509d5822461d761a62
    Stored in directory: /tmp/pip-ephem-wheel-cache-0inkbw7y/wheels/d4/a3/b2/6faade3bb04cd720bb432a6c21ec7baa2542801334a4d47a66
    Building wheel for setuptools (pyproject.toml): started
    Building wheel for setuptools (pyproject.toml): finished with status 'done'
    Created wheel for setuptools: filename=setuptools-70.0.0-py3-none-any.whl size=863432 sha256=5cb7584d1c8d0d8b8030486e5aa7b8580df190c717b4bddc06bcc9404239f3e9
    Stored in directory: /tmp/pip-ephem-wheel-cache-0inkbw7y/wheels/20/5f/b8/845eeb9e697798706931bd3ebc34b40c3bd3e3d6167407bb77
  Successfully built wheel setuptools
  WARNING: Ignoring invalid distribution -umpy (/home/mahmood/.local/lib/python3.8/site-packages)
  Installing collected packages: wheel, setuptools
  Successfully installed setuptools-70.0.0 wheel-0.43.0
  WARNING: Ignoring invalid distribution -umpy (/home/mahmood/.local/lib/python3.8/site-packages)
  WARNING: Ignoring invalid distribution -umpy (/home/mahmood/.local/lib/python3.8/site-packages)
  WARNING: Ignoring invalid distribution -umpy (/home/mahmood/.local/lib/python3.8/site-packages)
  WARNING: There was an error checking the latest version of pip.
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Traceback (most recent call last):
    File "/home/mahmood/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/mahmood/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/mahmood/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-8gsxes2v/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-8gsxes2v/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-8gsxes2v/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 5, in <module>
  ModuleNotFoundError: No module named 'packaging'
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/bin/python3 /home/mahmood/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpyowq808l
  cwd: /disk1/mahmood/training/large_language_model/megatron-lm/apex
  Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

But as you can see below, packaging exists:

$ pip3 list | grep packaging
packaging                         20.9
$ pip list | grep packaging
packaging                         20.9
$ dpkg -l | grep python3-packaging
ii  python3-packaging                                           20.3-1                                 all          core utilities for python3 packages
$ dpkg -l | grep python-packaging
ii  python-packaging                                            20.3-1                                 all          core utilities for python packages

Any idea about that?

[Li357]

Having the same issue, how'd you fix this?

[mahmoodn]

Actually a reboot fixed that, but I am not sure if that works for all cases...

[Wei-JL]

I have encountered the same problem and have installed 'packaging'. My system environment is aarch64 architecture with Huawei Ascend and the operating system is Ubuntu. How can I resolve the above issue?