Install with `poetry install` broken due to missing `pip` executable

carmenbianca commented 1 month ago

Details

Read the Docs project URL: https://readthedocs.org/projects/reuse
Build URL (if applicable): https://readthedocs.org/projects/reuse/builds/25578647/
Read the Docs username (if applicable): https://readthedocs.org/profiles/carmenbianca/

Expected Result

Using the following config:

build:
  os: ubuntu-22.04
  tools:
    python: "3.9"
  jobs:
    post_create_environment:
      - python -m pip install poetry
    post_install:
      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH python -m poetry install --with
        docs

The build should successfully use Poetry, as documented here

Actual Result

The build fails on the poetry install step.

The build also fails when removing the python -m substrings from the above configuration.

This is the output of the failed step:

Installing dependencies from lock file

Package operations: 49 installs, 9 updates, 0 removals

  - Downgrading certifi (2024.8.30 -> 2024.6.2)
  [... shortened for your convenience]
  - Installing sphinxcontrib-apidoc (0.5.0)

Installing the current project: reuse (4.0.3)
Preparing build environment with build-system requirements poetry-core>=1.1.0

Command ['/tmp/tmp9lc98fl8/.venv/bin/python', '/home/docs/checkouts/readthedocs.org/user_builds/reuse/envs/latest/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-24.2-py3-none-any.whl/pip', 'install', '--disable-pip-version-check', '--ignore-installed', '--no-input', 'poetry-core>=1.1.0'] errored with the following return code 2

Output:
/tmp/tmp9lc98fl8/.venv/bin/python: can't open file '/home/docs/checkouts/readthedocs.org/user_builds/reuse/envs/latest/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-24.2-py3-none-any.whl/pip': [Errno 2] No such file or directory

This started occurring at some point after 2024-09-05 (latest successful build), with 2024-09-10 as the earliest build that fails. There were no relevant changes to Sphinx, the RTD config, Poetry, dependencies, etc etc etc between these two dates.

Specifically, what appears to happen is this: As part of running poetry install, the build-system dependencies from pyproject.yaml must be installed. Snippet:

[build-system]
requires = ["poetry-core>=1.1.0"]
build-backend = "poetry.core.masonry.api"

As far as I can divine, Poetry (or Python?) delegates the installation of this dependency to Pip, and Pip then fails to install poetry-core because pip does not exist at a given path. I assume that:

Pip is either not installed in $READTHEDOCS_VIRTUALENV_PATH;
or it somehow divines a completely incorrect path where pip is (not) located.

carmenbianca commented 1 month ago

For reasons that are far beyond me, running poetry lock to re-generate poetry.lock has fixed the issue. See this PR: https://github.com/fsfe/reuse-tool/pull/1081

Some notes:

The commit of the last successful build did NOT build anymore when I attempted to build that commit again. Compare:
- https://readthedocs.org/projects/reuse/builds/25534005/ (last successful, 2024-09-05)
- https://readthedocs.org/projects/reuse/builds/25767550/ (same commit, failed, 2024-09-27)
- The commit in question
I used Poetry 1.3.x to generate the poetry.lock files.

I don't know if this is a bug on RTD's part. I will leave the issue open even though it is resolved for me, because this seems rather unusual to me, and the error message appears completely unrelated.

carmenbianca commented 2 days ago

This is still a bug.

humitos commented 2 days ago

You will able to use "partial overrides" when https://github.com/readthedocs/readthedocs.org/pull/11710 gets merged, with something like the following:

build:
  os: ubuntu-22.04
  tools:
    python: "3.9"
  jobs:
    create_environment:
      - python -m pip install poetry
    install:
      - python -m poetry install --with docs

readthedocs / readthedocs.org