search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.29k stars 628 forks source link

Lockfile generation failed due to "ModuleNotFoundError: No module named 'numpy'" #21115

Open jasondamour opened 3 months ago

jasondamour commented 3 months ago

Describe the bug Reproduction Repository: https://github.com/jasondamour/pants-numpy-issue-repro When trying to generate a lockfile for a project depending on gensim, lockfile fails:

➜  pants generate-lockfiles --resolve=test                                
15:59:04.61 [INFO] Completed: Generate lockfile for test
15:59:04.61 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for test' failed with exit code 1.
stdout:

stderr:
Could not gather lock metadata for 1 project with source artifacts:
1. /private/var/folders/_9/qvrc0kmj4ssfvbybnz4_wb7c0000gn/T/pants-sandbox-ia6HBt/.tmp/tmp9ajiuo7d/[redact]..pyenv.versions.3.10.6.bin.python3.10/gensim-3.8.0.tar.gz: Executing $HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/bin/python -sE $HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/pex -c import json
import sys

import setuptools.build_meta

if not hasattr(setuptools.build_meta.__legacy__, 'prepare_metadata_for_build_wheel'):
    sys.exit(75)

result = setuptools.build_meta.__legacy__.prepare_metadata_for_build_wheel(*('/private/var/folders/_9/qvrc0kmj4ssfvbybnz4_wb7c0000gn/T/pants-sandbox-ia6HBt/.tmp/tmpbbcrhe_3/build',), **{})
with open('/private/var/folders/_9/qvrc0kmj4ssfvbybnz4_wb7c0000gn/T/pants-sandbox-ia6HBt/.tmp/tmp_ac1boah', "w") as fp:
    json.dump(result, fp)
 failed with 1

STDERR:
Traceback (most recent call last):
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/pex", line 311, in <module>
    exec(ast, globals_map, locals_map)
  File "-c <cmd>", line 10, in <module>
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/build_meta.py", line 380, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/build_meta.py", line 487, in run_setup
    super(_BuildMetaLegacyBackend,
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
    exec(code, locals())
  File "<string>", line 334, in <module>
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/command/dist_info.py", line 99, in run
    self.egg_info.run()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 317, in run
    self.find_sources()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 325, in find_sources
    mm.run()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 573, in run
    self.add_defaults()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 611, in add_defaults
    sdist.add_defaults(self)
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/command/sdist.py", line 106, in add_defaults
    super().add_defaults()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
    self._add_defaults_ext()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
    build_ext = self.get_finalized_command('build_ext')
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 305, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "$HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
    self.finalize_options()
  File "<string>", line 100, in finalize_options
ModuleNotFoundError: No module named 'numpy'

Pants version 2.20

OS Macos, haven't tested linux

Additional info Gensim is clearly doing some wonky stuff with numpy: https://github.com/piskvorky/gensim/blob/3.8.3/setup.py

Looking into the pants cache, numpy is not present (as expected by pants):

➜  ls $HOME/.cache/pants/named_caches/pex_root/venvs/8def0dffa12ba607a88711ae9e9538026fd843d0/4c982503433997e579e4274163218f2997dc553e/lib/python3.10/site-packages/            
PEX_EXTRA_SYS_PATH.pth      distutils-precedence.pth    setuptools                  wheel
_distutils_hack             pkg_resources               setuptools-67.7.2.dist-info wheel-0.40.0.dist-info
jsirois commented 2 months ago

@jasondamour your suspicion is correct. The problem is specifically with the use of the setuptools.setup setup_requires argument which was a big part of the motivation for PEP-518. See here for the same problem and a self serve solution: https://github.com/pantsbuild/pants/issues/19898#issuecomment-1730109755