Projects with extension modules and requirements from private repository no longer installed

[Lasall]

• Poetry version: 1.6.1
• Python version: 3.11.5
• OS version and name: Debian sid
• pyproject.toml: https://github.com/Lasall/poetry-editable-extensions/blob/main/pyproject.toml

• [x] I am on the latest stable Poetry version, installed using a recommended method.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have consulted the FAQ and blog for any relevant entries or release notes.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Similar to https://github.com/python-poetry/poetry/issues/8377 I experience some issues after https://github.com/python-poetry/poetry/pull/7975 .

In my setup I can only use a private PyPi repo, access to pypi.org is not possible. Therefore I've configured a poetry default-priority source which works just fine. However when using an extension module starting with poetry version 1.6 it will actually try to fetch the build requirements (from pypi.org). Since the pip-command is invoked with --isolate I don`t see a way to make it use my private repo for the build requirements as well.

Example to reproduce this setup: https://github.com/Lasall/poetry-editable-extensions

$ poetry install -vvv
Adding repository Local repo (http://127.0.0.1:9000) and setting it as the default one
Deactivating the PyPI repository
Using virtualenv: /tmp/poetry-editable-extensions/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 0 updates, 0 removals, 1 skipped

  • Installing pybind11 (2.10.4): Pending...
  • Installing pybind11 (2.10.4): Skipped for the following reason: Already installed

Installing the current project: poetry-editable-extensions (0.1.0)
  - Building package poetry-editable-extensions in editable mode
  - Falling back on using a setup.py
[...]
  Stack trace:

  2  .venv/lib/python3.11/site-packages/poetry/utils/env/base_env.py:359 in _run
      357│                 output = ""
      358│             else:
    → 359│                 output = subprocess.check_output(
      360│                     cmd, stderr=stderr, env=env, text=True, **kwargs
      361│                 )

  1  /usr/lib/python3.11/subprocess.py:466 in check_output
       464│         kwargs['input'] = empty
       465│ 
    →  466│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       467│                **kwargs).stdout
       468│ 

  CalledProcessError

  Command '['/tmp/poetry-editable-extensions/.venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/tmp/poetry-editable-extensions/.venv', '--upgrade', '--no-deps', '-e', '/tmp/poetry-editable-extensions']' returned non-zero exit status 1.

  at /usr/lib/python3.11/subprocess.py:571 in run
       567│             # We don't call process.wait() as .__exit__ does that for us.
       568│             raise
       569│         retcode = process.poll()
       570│         if check and retcode:
    →  571│             raise CalledProcessError(retcode, process.args,
       572│                                      output=stdout, stderr=stderr)
       573│     return CompletedProcess(process.args, retcode, stdout, stderr)
       574│ 
       575│ 

The following error occurred when trying to handle this error:

  Stack trace:

  3  .venv/lib/python3.11/site-packages/poetry/utils/pip.py:56 in pip_install
       54│ 
       55│     try:
    →  56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
       58│         raise PoetryException(f"Failed to install {path}") from e

  2  .venv/lib/python3.11/site-packages/poetry/utils/env/base_env.py:319 in run_pip
      317│         pip = self.get_pip_command()
      318│         cmd = pip + list(args)
    → 319│         return self._run(cmd, **kwargs)
      320│ 
      321│     def run_python_script(self, content: str, **kwargs: Any) -> str:

  1  .venv/lib/python3.11/site-packages/poetry/utils/env/virtual_env.py:96 in _run
       94│     def _run(self, cmd: list[str], **kwargs: Any) -> str:
       95│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    →  96│         return super()._run(cmd, **kwargs)
       97│ 
       98│     def get_temp_environ(

  EnvCommandError

  Command ['/tmp/poetry-editable-extensions/.venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/tmp/poetry-editable-extensions/.venv', '--upgrade', '--no-deps', '-e', '/tmp/poetry-editable-extensions'] errored with the following return code 1

  Output:
  Obtaining file:///tmp/poetry-editable-extensions
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    error: subprocess-exited-with-error

    × pip subprocess to install build dependencies did not run successfully.
    │ exit code: 1
    ╰─> [7 lines of output]
        WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ff22668e610>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/poetry-core/
        WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ff22668f610>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/poetry-core/
        WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ff22668fe50>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/poetry-core/
        WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ff226698990>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/poetry-core/
        WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ff226699550>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/poetry-core/
        ERROR: Could not find a version that satisfies the requirement poetry-core (from versions: none)
        ERROR: No matching distribution found for poetry-core
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.

  at .venv/lib/python3.11/site-packages/poetry/utils/env/base_env.py:363 in _run
      359│                 output = subprocess.check_output(
      360│                     cmd, stderr=stderr, env=env, text=True, **kwargs
      361│                 )
      362│         except CalledProcessError as e:
    → 363│             raise EnvCommandError(e, input=input_)
      364│ 
      365│         return output
      366│ 
      367│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:

  Stack trace:

  10  .venv/lib/python3.11/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

   9  .venv/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

   8  .venv/lib/python3.11/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

   7  .venv/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

   6  .venv/lib/python3.11/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   5  .venv/lib/python3.11/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   4  .venv/lib/python3.11/site-packages/cleo/commands/command.py:62 in execute
        60│ 
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

   3  .venv/lib/python3.11/site-packages/poetry/console/commands/install.py:181 in handle
       179│             return 0
       180│ 
     → 181│         builder.build()
       182│ 
       183│         if overwrite:

   2  .venv/lib/python3.11/site-packages/poetry/masonry/builders/editable.py:63 in build
        61│                     "  - Falling back on using a setup.py"
        62│                 )
     →  63│                 self._setup_build()
        64│                 path: Path = self._path
        65│                 return path

   1  .venv/lib/python3.11/site-packages/poetry/masonry/builders/editable.py:104 in _setup_build
       102│ 
       103│         try:
     → 104│             pip_install(self._path, self._env, upgrade=True, editable=True)
       105│         finally:
       106│             if not has_setup:

  PoetryException

  Failed to install /tmp/poetry-editable-extensions

  at .venv/lib/python3.11/site-packages/poetry/utils/pip.py:58 in pip_install
       54│ 
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path}") from e
       59│ 

[dimbleby]

prior to #7975 poetry completely ignored build requirements so you couldn't build anyway: this is progress!

preferred fix should be to avoid using pip at all and do more or less what Chef.prepare() does (preferably reusing as much code as possible). That class already understands how to make use of poetry's pool of repositories.

if this is important to you... pull requests surely welcome...

[Lasall]

As a workaround it is possible to define the PIP_INDEX_URL not as environment variable (which is ignored with pip --isolate) but write it to global config /etc/pip.conf:

[global]
index-url = ...