Support for wheel dependency vendoring

[danieljanes]

Some tools support vendoring of dependencies by depending on local .whl files (as opposed to copying the source files, example: https://python-poetry.org/docs/dependency-specification/#path-dependencies). The motivation for doing so would be more reliable builds and no necessity for accessing external services during build time / from CI.

Are there plans to support this in pants?

[jsirois]

Pants will support this as soon as #3063 is fixed. In short, at that point Pants will support specifying any requirement string Pip supports and Pip does support specifying local distributions in at least 2 ways:

1. Via a relative or absolute path to the archive
2. Via a "Direct reference": https://www.python.org/dev/peps/pep-0440/#direct-references

Related is also #8669 which needs the same fix.

In the meantime, you can also achieve this - even on an ad-hoc basis - by using "normal" requirments strings but altering how Pants searches for distributions (You can read more digestable information about the options and concepts / files used bwelow here: https://www.pantsbuild.org/docs/python-third-party-dependencies):

$ ./pants help-advanced python-repos

`python-repos` subsystem options
--------------------------------

External Python code repositories, such as PyPI.

No options available.

`python-repos` subsystem advanced options
-----------------------------------------

  --python-repos-indexes="['<str>', '<str>', ...]"
      default: [
          "https://pypi.org/simple/"
      ]
      current value: [
          "https://pypi.org/simple/"
      ]
      URLs of code repository indexes to look for requirements. If set to an empty list, then Pex will
      use no indices (meaning it will not use PyPI). The values should be compliant with PEP 503.

  --python-repos-repos="['<str>', '<str>', ...]"
      default: []
      current value: []
      URLs of code repositories to look for requirements. In Pip and Pex, this option corresponds to
      the `--find-links` option.

`python-repos` subsystem deprecated options
-------------------------------------------

No options available.

The following session demonstrates this in action by turning off PyPI and turning on a local distribution directory:

$ curl -sSL https://files.pythonhosted.org/packages/1d/b4/b008b7ec1fc3b60923277c955de488d533613f0ac4338d89b2cf353ac5ae/isort-5.1.4-py3-none-any.whl > issues-10446/repo/isort-5.1.4-py3-none-any.whl
$ cat issues-10446/3rdparty/python/requirements.txt 
isort>5

$ cat issues-10446/3rdparty/python/BUILD 
python_requirements()

$ cat issues-10446/src/python/BUILD 
python_binary(
    name="isort",
    dependencies=[
        "issues-10446/3rdparty/python:isort",
    ],
    entry_point="isort",
)

$ ./pants --python-repos-repos="['"$PWD/issues-10446/repo"']" --python-repos-indexes="[]" run issues-10446/src/python:isort
15:03:26.02 [WARN] The constraints file 3rdparty/python/requirements.txt does not contain entries for the following requirements: isort

                 _                 _
                (_) ___  ___  _ __| |_
                | |/ _/ / _ \/ '__  _/
                | |\__ \/\_\/| |  | |_
                |_|\___/\___/\_/   \_/

      isort your imports, so you don't have to.

                    VERSION 5.1.4

Nothing to do: no files or paths have have been passed in!

Try one of the following:

    `isort .` - sort all Python files, starting from the current directory, recursively.
    `isort . --interactive` - Do the same, but ask before making any changes.
    `isort . --check --diff` - Check to see if imports are correctly sorted within this project.
    `isort --help` - In-depth information about isort's available command-line options.

Visit https://timothycrosley.github.io/isort/ for complete information about how to use isort.

And to prove this is actually using the local wheel you can eliminate the custom local repo --python-repos-repos="['"$PWD/issues-10446/repo"']" and observe:

$ ./pants --python-repos-indexes="[]" run issues-10446/src/python:isort
15:03:15 [INFO] initialization options changed: reinitializing pantsd...
15:03:16 [INFO] pantsd initialized.
15:03:16.71 [WARN] The constraints file 3rdparty/python/requirements.txt does not contain entries for the following requirements: isort
15:03:17 [ERROR] 1 Exception encountered:

Engine traceback:
  in `run` goal
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/process.py", line 215, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building isort.pex with 1 requirement: isort>5' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement isort>5 (from versions: none)
ERROR: No matching distribution found for isort>5
pid: 81920 -> /home/jsirois/dev/pantsbuild/jsirois-pants/build-support/virtualenvs/Linux/pants_dev_deps.py38.venv/bin/python3 /home/jsirois/.cache/pants/named_caches/pex_root/pip.pex/93588224cef361f7437ddb8f30d12161967d077e --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root download --dest /tmp/tmp_va3dg_7/home.jsirois.dev.pantsbuild.jsirois-pants.build-support.virtualenvs.Linux.pants_dev_deps.py38.venv.bin.python3 --no-index --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --constraint 3rdparty/python/requirements.txt isort>5 raised Executing /home/jsirois/dev/pantsbuild/jsirois-pants/build-support/virtualenvs/Linux/pants_dev_deps.py38.venv/bin/python3 /home/jsirois/.cache/pants/named_caches/pex_root/pip.pex/93588224cef361f7437ddb8f30d12161967d077e --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root download --dest /tmp/tmp_va3dg_7/home.jsirois.dev.pantsbuild.jsirois-pants.build-support.virtualenvs.Linux.pants_dev_deps.py38.venv.bin.python3 --no-index --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --constraint 3rdparty/python/requirements.txt isort>5 failed with 1

Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 288, in run
    engine_result = self._run_v2()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 176, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 193, in _maybe_run_v2_body
    return self.graph_session.run_goal_rules(
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/init/engine_initializer.py", line 134, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/scheduler.py", line 558, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/scheduler.py", line 517, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in `run` goal
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/process.py", line 215, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building isort.pex with 1 requirement: isort>5' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement isort>5 (from versions: none)
ERROR: No matching distribution found for isort>5
pid: 81920 -> /home/jsirois/dev/pantsbuild/jsirois-pants/build-support/virtualenvs/Linux/pants_dev_deps.py38.venv/bin/python3 /home/jsirois/.cache/pants/named_caches/pex_root/pip.pex/93588224cef361f7437ddb8f30d12161967d077e --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root download --dest /tmp/tmp_va3dg_7/home.jsirois.dev.pantsbuild.jsirois-pants.build-support.virtualenvs.Linux.pants_dev_deps.py38.venv.bin.python3 --no-index --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --constraint 3rdparty/python/requirements.txt isort>5 raised Executing /home/jsirois/dev/pantsbuild/jsirois-pants/build-support/virtualenvs/Linux/pants_dev_deps.py38.venv/bin/python3 /home/jsirois/.cache/pants/named_caches/pex_root/pip.pex/93588224cef361f7437ddb8f30d12161967d077e --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root download --dest /tmp/tmp_va3dg_7/home.jsirois.dev.pantsbuild.jsirois-pants.build-support.virtualenvs.Linux.pants_dev_deps.py38.venv.bin.python3 --no-index --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --constraint 3rdparty/python/requirements.txt isort>5 failed with 1

[danieljanes]

Thanks a lot for your detailed answer @jsirois ! I wasn't aware of that feature / workaround :)

[Eric-Arellano]

Sounds like this is stale, as Pants now supports PEP 440.

[jsirois]

Pants will support this as soon as #3063 is fixed.

This assertion was false. To support a vendored wheel checked in to a repo Pants needs to support both relative path requirements and a sane way to depend on those requirements such that they get carried along into Process sandboxes with the appropriate relative path. You can probably arrange this today with a file dependency? but it's more than a stretch to call this "pants supports vendoring".

[jsirois]

To make the above a bit more concrete, using the example-python repo, you can vendor a wheel like so:

$ git diff --cached
diff --git a/requirements.txt b/requirements.txt
index ffab1be..ea93824 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,7 +1,7 @@
 # Copyright 2020 Pants project contributors.
 # Licensed under the Apache License, Version 2.0 (see LICENSE).

-ansicolors>=1.0.2
+ansicolors @ file:///home/jsirois/dev/pantsbuild/example-python/vendored/ansicolors-1.1.8-py2.py3-none-any.whl
 setuptools<54.0,>=50.3.0
 translate>=3.2.1
 protobuf>=3.11.3
diff --git a/vendored/ansicolors-1.1.8-py2.py3-none-any.whl b/vendored/ansicolors-1.1.8-py2.py3-none-any.whl
new file mode 100644
index 0000000..e9619aa
Binary files /dev/null and b/vendored/ansicolors-1.1.8-py2.py3-none-any.whl differ

The problem here is that the file: URL must be absolute to parse via packaging / pkg_resources / Pip. Clearly checking in a requirements file with an absolute path like that is a non-starter. We'd need to support something like:

ansicolors @ file://$PWD/vendored/ansicolors-1.1.8-py2.py3-none-any.whl

Or:

ansicolors @ file://$BUILDROOT/vendored/ansicolors-1.1.8-py2.py3-none-any.whl

And ensure the corresponding env var was exported when the resolve Process is executed. The env var could point to the path of the clone with a loss in hermeticity or it could point to the Process sandbox if we also added the vendored wheel as a file dependency by eliminating it from requirements.txt and instead ginning up something like this in a top-level BUILD file:

files(
  name="_vendored_ansicolors_wheel",
  sources=[
    "vendored/ansicolors-1.1.8-py2.py3-none-any.whl",
  ],
)

python_requirement_library(
  name="ansicolors",
  requirements=[
    "file://$PWD/vendored/ansicolors-1.1.8-py2.py3-none-any.whl",
  ],
  dependencies=[
    ":_vendored_ansicolors_wheel",
  ],
)

If Pants passed requirements to Pex always via synthetic requirements files (which is being worked on ~currently for other reasons); then this would all just work since Pex, like Pip, supports env var interpolation in requirements files. All Pants would need to do is export the appropriate env var in the Pex resolve Process.

[thejcannon]

@Eric-Arellano can we close this as, if I'm not mistaken, this is supported in the new PEX lockfiles?

[jsirois]

It's supported by Pex with --path-mapping but Pants does nothing with this feature yet.

[stuhood]

@Eric-Arellano : You integrated path-mapping, right?

[jsirois]

They did, but we still can't support ansicolors @ file://$PWD/vendored/ansicolors-1.1.8-py2.py3-none-any.whl in the originating requirements file IIUC. We have toml support for doing something like this but neither BUILD support, nor support for Pip requirement parsing behavior which respects env vars. Pex supports this, Pants does not.

[Eric-Arellano]

My understanding was that Pants is okay with not robustly supporting the file:// format where you can't use an absolute path. Insted, you can use [python-repos].find_links