setuptools 50.3.0 makes build fail

[AchmadFathoni]

I have tried to compile ros2 rolling on my Arch Linux but got this error:

Starting >>> ament_package
Starting >>> ament_lint
Starting >>> gtest_vendor
Starting >>> fastcdr
Starting >>> ament_cppcheck
Starting >>> cyclonedds
Starting >>> osrf_pycommon
Starting >>> osrf_testing_tools_cpp
Traceback (most recent call last):] [ament_package - 0.1s] [ament_lint - 0.1s] [gtest_vendor - 0.1s] ...
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools.extern'
--- stderr: ament_package                                                                               
Traceback (most recent call last):
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_ros/package_identification/ros.py", line 129, in getter
    return get_setup_information(
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 261, in get_setup_information
    _setup_information_cache[hashable_env] = _get_setup_information(
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 301, in _get_setup_information
    result = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.
---
Failed   <<< ament_package [0.19s, exited with code 1]
Traceback (most recent call last): [7 ongoing] [ament_lint - 0.3s] [gtest_vendor - 0.3s] [fastcdr - 0.3s] ...
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools.extern'
--- stderr: ament_lint                                                                                       
Traceback (most recent call last):
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_ros/package_identification/ros.py", line 129, in getter
    return get_setup_information(
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 261, in get_setup_information
    _setup_information_cache[hashable_env] = _get_setup_information(
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 301, in _get_setup_information
    result = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.
---
Failed   <<< ament_lint [0.37s, exited with code 1]
Traceback (most recent call last): [6 ongoing] [gtest_vendor:build - 0.5s] [fastcdr:cmake - 0.5s] ...
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools.extern'
--- stderr: ament_cppcheck                                                                           
Traceback (most recent call last):
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_ros/package_identification/ros.py", line 129, in getter
    return get_setup_information(
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 261, in get_setup_information
    _setup_information_cache[hashable_env] = _get_setup_information(
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 301, in _get_setup_information
    result = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.
---
Failed   <<< ament_cppcheck [0.57s, exited with code 1]
Traceback (most recent call last): [5 ongoing] [gtest_vendor:build - 0.7s] [fastcdr:cmake - 0.7s] ...
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools.extern'
--- stderr: osrf_pycommon                                                                            
Traceback (most recent call last):
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_ros/package_identification/ros.py", line 129, in getter
    return get_setup_information(
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 261, in get_setup_information
    _setup_information_cache[hashable_env] = _get_setup_information(
  File "/home/toni/.local/lib/python3.8/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 301, in _get_setup_information
    result = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.
---
Failed   <<< osrf_pycommon [0.79s, exited with code 1]
Aborted  <<< gtest_vendor [0.80s]
Aborted  <<< osrf_testing_tools_cpp [0.79s]
Aborted  <<< fastcdr [0.80s]
Aborted  <<< cyclonedds [0.80s]

Summary: 0 packages finished [4.13s]
  4 packages failed: ament_cppcheck ament_lint ament_package osrf_pycommon
  4 packages aborted: cyclonedds fastcdr gtest_vendor osrf_testing_tools_cpp
  4 packages had stderr output: ament_cppcheck ament_lint ament_package osrf_pycommon
  297 packages not processed
Command '['/usr/bin/python', '-c', "import sys;from setuptools.extern.packaging.specifiers import SpecifierSet;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1.

Fixed by changing python_setup_py.py line 251: 'from setuptools.extern.packaging.specifiers import SpecifierSet', to 'from packaging.specifiers import SpecifierSet',

[dirk-thomas]

Please do not cross post: https://answers.ros.org/question/362883/fail-building-foxy-modulenotfounderror-no-module-named-setuptoolsextern/ Especially without referencing the duplicate posts.

[AchmadFathoni]

Well, somehow installing setuptools from arch package manager doesn't include extern sub-package. Here is the content of /usr/lib/python3.8/site-packages/setuptools which missing extern

archive_util.py           dist.py        installer.py       py34compat.py        version.py
build_meta.py             _distutils     launch.py          __pycache__          wheel.py
command                   errors.py      lib2to3_ex.py      sandbox.py           windows_support.py
config.py                 extension.py   monkey.py         'script (dev).tmpl'
depends.py                glob.py        msvc.py            script.tmpl
_deprecation_warning.py   _imp.py        namespaces.py      ssl_support.py
dep_util.py               __init__.py    package_index.py   unicode_utils.py

Here is my setuptools that I installed from virtual environment from /.local/share/virtualenvs/ros2_foxy-8tYMXHqm/lib/python3.8/site-packages/setuptools which has extern.

 archive_util.py   _deprecation_warning.py   gui-32.exe      monkey.py            ssl_support.py
 build_meta.py     dep_util.py               gui-64.exe      msvc.py              unicode_utils.py
 cli-32.exe        dist.py                   gui.exe         namespaces.py        _vendor
 cli-64.exe        _distutils                _imp.py         package_index.py     version.py
 cli.exe           errors.py                 __init__.py     py34compat.py        wheel.py
 command           extension.py              installer.py    sandbox.py           windows_support.py
 config.py         extern                    launch.py      'script (dev).tmpl'
 depends.py        glob.py                   lib2to3_ex.py   script.tmpl

I am done, you can close my issue and pull request now.

[dirk-thomas]

somehow installing setuptools from arch package manager doesn't include extern sub-package.

@AchmadFathoni Can you file a ticket with Arch about this problem and post the link here for future readers?

[AchmadFathoni]

somehow installing setuptools from arch package manager doesn't include extern sub-package.

@AchmadFathoni Can you file a ticket with Arch about this problem and post the link here for future readers?

This is Arch's python-setuptools PKGBUILD. I don't understands advanced bash script. If you know bash @dirk-thomas, do you know where they remove extern package? and what python line should I use?

prepare() {
  rm -r setuptools-$pkgver/{pkg_resources,setuptools}/{extern,_vendor}

  # Upstream devendoring logic is badly broken, see:
  # https://bugs.archlinux.org/task/58670
  # https://github.com/pypa/pip/issues/5429
  # https://github.com/pypa/setuptools/issues/1383
  # The simplest fix is to simply rewrite import paths to use the canonical
  # location in the first place
  for _module in setuptools pkg_resources '' ; do
      find setuptools-$pkgver -name \*.py -exec sed -i \
          -e 's/from '$_module.extern' import/import/' \
          -e 's/from '$_module.extern'./from /' \
          -e 's/import '$_module.extern'./import /' \
          -e "s/__import__('$_module.extern./__import__('/" \
          {} +
    done

[dirk-thomas]

do you know where they remove extern package?

rm -r setuptools-$pkgver/{pkg_resources,setuptools}/{extern,_vendor}

[AchmadFathoni]

do you know where they remove extern package?

rm -r setuptools-$pkgver/{pkg_resources,setuptools}/{extern,_vendor}

So according to that PKGBUILD. what line should is use to import extern now? Or specifiacally Specifier object.

[dirk-thomas]

I am sorry, I don't understand the question.

[AchmadFathoni]

I am sorry, I don't understand the question.

Sorry I just woke up and it is 3AM here. Let me repeat my question. So according to that PKGBUILD, what line should I is use to import extern package now? Or specially Specifier object.

[AchmadFathoni]

Anyway, This is my thread on Arch's newbie corner. The post number 5 explains a lot about this problem.

[dirk-thomas]

I don't know anything about PKGBUILD. I would assume removing the line starting with rm -r would be a start.

This is my thread on Arch's newbie corner. The post number 5 explains a lot about this problem.

It sounds like Arch did drop those modules intentionally and doesn't want to readd them.

Please see my comment / question on https://github.com/colcon/colcon-python-setup-py/pull/42#issuecomment-704408599 to instead try a fix I proposed.

[romain130492]

Hello what's the solution for ros2 humble? It's still happening apparently