Attempts to install build tools to system Python environment

[jaraco]

Using pip-run to invoke fromager, it gets a lot done before attempting to install setuptools to the system Python environment and then fails because pip rejects the installation:

 fromager 🐚 py -3.12 -m pip-run fromager -- -m fromager bootstrap requests
bootstrapping 'cpu' variant of [('toplevel', 'requests')]
requests: * handling toplevel requirement requests []
saved /Users/jaraco/fromager/sdists-repo/downloads/requests-2.32.3.tar.gz
requests: new toplevel dependency requests resolves to 2.32.3
requests: preparing source for requests from /Users/jaraco/fromager/sdists-repo/downloads/requests-2.32.3.tar.gz
requests: prepared source for requests at /Users/jaraco/fromager/work-dir/requests-2.32.3/requests-2.32.3
requests: getting build system dependencies for requests in /Users/jaraco/fromager/work-dir/requests-2.32.3/requests-2.32.3
setuptools: ** handling build-system requirement setuptools>=40.8.0 [('toplevel', <Requirement('requests')>, <Version('2.32.3')>)]
saved /Users/jaraco/fromager/sdists-repo/downloads/setuptools-71.0.3.tar.gz
setuptools: new build-system dependency setuptools>=40.8.0 resolves to 71.0.3
setuptools: preparing source for setuptools>=40.8.0 from /Users/jaraco/fromager/sdists-repo/downloads/setuptools-71.0.3.tar.gz
setuptools: prepared source for setuptools>=40.8.0 at /Users/jaraco/fromager/work-dir/setuptools-71.0.3/setuptools-71.0.3
setuptools: getting build system dependencies for setuptools>=40.8.0 in /Users/jaraco/fromager/work-dir/setuptools-71.0.3/setuptools-71.0.3
setuptools: getting build backend dependencies for setuptools>=40.8.0 in /Users/jaraco/fromager/work-dir/setuptools-71.0.3/setuptools-71.0.3
setuptools: getting build sdist dependencies for setuptools>=40.8.0 in /Users/jaraco/fromager/work-dir/setuptools-71.0.3/setuptools-71.0.3
setuptools: adding ('setuptools', '71.0.3') to build order
setuptools: preparing to build wheel for version 71.0.3
created build environment in /Users/jaraco/fromager/work-dir/setuptools-71.0.3/build-3.12.4
setuptools: building source distribution in /Users/jaraco/fromager/work-dir/setuptools-71.0.3/setuptools-71.0.3
setuptools: built source distribution /Users/jaraco/fromager/sdists-repo/builds/setuptools-71.0.3.tar.gz
setuptools: building wheel for setuptools>=40.8.0 in /Users/jaraco/fromager/work-dir/setuptools-71.0.3/setuptools-71.0.3 writing to /Users/jaraco/fromager/wheels-repo/build
setuptools: built wheel for version 71.0.3: /Users/jaraco/fromager/wheels-repo/downloads/setuptools-71.0.3-py3-none-any.whl
setuptools: getting installation dependencies from /Users/jaraco/fromager/wheels-repo/downloads/setuptools-71.0.3-py3-none-any.whl
['/opt/homebrew/Cellar/python@3.12/3.12.4/bin/python3.12', '-m', 'pip', '-vvv', 'install', '--disable-pip-version-check', '--upgrade', '--only-binary', ':all:', '--index-url', 'http://localhost:58655/simple/', '--trusted-host', 'localhost', 'setuptools>=40.8.0'] failed with error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:

    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz

    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with

    brew install pipx

    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.

    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.

    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
Exception information:
Traceback (most recent call last):
  File "/Users/jaraco/.local/pip-run/pip/_internal/cli/base_command.py", line 179, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/Users/jaraco/.local/pip-run/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jaraco/.local/pip-run/pip/_internal/commands/install.py", line 285, in run
    check_externally_managed()
  File "/Users/jaraco/.local/pip-run/pip/_internal/utils/misc.py", line 615, in check_externally_managed
    raise ExternallyManagedEnvironment.from_config(marker)
pip._internal.exceptions.ExternallyManagedEnvironment: <ExternallyManagedEnvironment: externally-managed-environment>

Command '['/opt/homebrew/Cellar/python@3.12/3.12.4/bin/python3.12', '-m', 'pip', '-vvv', 'install', '--disable-pip-version-check', '--upgrade', '--only-binary', ':all:', '--index-url', 'http://localhost:58655/simple/', '--trusted-host', 'localhost', 'setuptools>=40.8.0']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/__main__.py", line 156, in invoke_main
    main(auto_envvar_prefix="FROMAGER")
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/commands/bootstrap.py", line 63, in bootstrap
    sdist.handle_requirement(wkctx, Requirement(toplevel), req_type=origin)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/sdist.py", line 157, in handle_requirement
    build_system_dependencies = _handle_build_system_requirements(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/sdist.py", line 340, in _handle_build_system_requirements
    _maybe_install(ctx, dep, "build-system", resolved)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/sdist.py", line 506, in _maybe_install
    safe_install(ctx, req, req_type)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/sdist.py", line 515, in safe_install
    external_commands.run(
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/external_commands.py", line 50, in run
    raise subprocess.CalledProcessError(completed.returncode, cmd, output)
subprocess.CalledProcessError: Command '['/opt/homebrew/Cellar/python@3.12/3.12.4/bin/python3.12', '-m', 'pip', '-vvv', 'install', '--disable-pip-version-check', '--upgrade', '--only-binary', ':all:', '--index-url', 'http://localhost:58655/simple/', '--trusted-host', 'localhost', 'setuptools>=40.8.0']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/__main__.py", line 163, in <module>
    invoke_main()
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/__main__.py", line 156, in invoke_main
    main(auto_envvar_prefix="FROMAGER")
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/commands/bootstrap.py", line 63, in bootstrap
    sdist.handle_requirement(wkctx, Requirement(toplevel), req_type=origin)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/sdist.py", line 157, in handle_requirement
    build_system_dependencies = _handle_build_system_requirements(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/sdist.py", line 340, in _handle_build_system_requirements
    _maybe_install(ctx, dep, "build-system", resolved)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/sdist.py", line 506, in _maybe_install
    safe_install(ctx, req, req_type)
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/sdist.py", line 515, in safe_install
    external_commands.run(
  File "/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-run-wlv7pq34/fromager/external_commands.py", line 50, in run
    raise subprocess.CalledProcessError(completed.returncode, cmd, output)
subprocess.CalledProcessError: Command '['/opt/homebrew/Cellar/python@3.12/3.12.4/bin/python3.12', '-m', 'pip', '-vvv', 'install', '--disable-pip-version-check', '--upgrade', '--only-binary', ':all:', '--index-url', 'http://localhost:58655/simple/', '--trusted-host', 'localhost', 'setuptools>=40.8.0']' returned non-zero exit status 1.

Perhaps the instructions should indicate that fromager should (must?) be run from a virtualenv as it attempts to modify the environment in which it runs.

Even better would be for fromager to use a tool like virtualenv or pip-run to install these dependencies on-demand rather than relying on the caller to set up a mutable environment.

[dhellmann]

We're using it in a container, so we haven't hit this particular issue.

We can at least update the docs to say you need to run in a virtualenv. I'm not sure what would be involved in reworking the build backend dependency execution to extract dependencies. It may be possible to set up an isolated virtualenv for that.