pex-tool / pex

A tool for generating .pex (Python EXecutable) files, lock files and venvs.
https://docs.pex-tool.org/
Apache License 2.0
2.53k stars 259 forks source link

unable to create pex with --no-build flag from lockfile #2389

Closed cburroughs closed 6 months ago

cburroughs commented 6 months ago

Example:

$ pex3 --version
2.2.2
$ pex3 lock create black==22.8.0 -o black.lock --style=universal --pip-version=23.3.2 --resolver-version pip-2020-resolver --indent=2 --interpreter-constraint='CPython==3.10.*' --wheel --no-build
$ pex --tmpdir .tmp --jobs 1 --no-emit-warnings --pip-version 23.3.2 --output-file black.pex --venv prepend --seed verbose --venv-site-packages-copies --python /usr/bin/python3.10 --console-script black  $'black==22.8.0' --lock=black.lock  --manylinux manylinux2014 --wheel --no-build   --layout packed -vvv
pex: Resolving interpreters: 2.0ms                        
pex:   Searching for pyenv root...: 0.1ms
pex: Parsing lock black.lock: 5.0ms
pex: Building pex :: Resolving distributions for requirements: black==22.8.0 :: Resolving requirements from lock file black.lock :: pex: Building pex :: Resolving distributions for requirements: black==22.8.0 :: Resolving requirements from lock file black.lock :: pex: Building pex :: Resolving distributions for requirements: black==22.8.0 :: Resolving requirements from lock file black.lock :: pex: Building pex :: Resolving distributions for requirements: black==22.8.0 :: Resolving requirements from lock file black.lock :: Traceback (most recent call last):
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/bin/pex.py", line 1045, in do_main
    pex_builder = build_pex(
                  ^^^^^^^^^^
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/bin/pex.py", line 912, in build_pex
    resolve(
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/resolve/configured_resolve.py", line 42, in resolve
    resolve_from_lock(
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/resolve/lock_resolver.py", line 300, in resolve_from_lock
    vcs_download_managers_by_target = {
                                      ^
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/resolve/lock_resolver.py", line 301, in <dictcomp>
    resolved_subset.target: VCSArtifactDownloadManager(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/resolve/lock_resolver.py", line 104, in __init__
    self._build_configuration = attr.evolve(
                                ^^^^^^^^^^^^
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/vendor/_vendored/attrs/attr/_funcs.py", line 370, in evolve
    return cls(**changes)
           ^^^^^^^^^^^^^^
  File "<attrs generated init pex.resolve.resolver_configuration.BuildConfiguration>", line 10, in __init__
    self.__attrs_post_init__()
  File "/home/ecsb/.local/lib/python3.11/site-packages/pex/resolve/resolver_configuration.py", line 124, in __attrs_post_init__
    raise self.Error(
pex.resolve.resolver_configuration.BuildConfiguration.Error: Cannot both disallow builds and disallow wheels. Please allow one of these or both so that some distributions can be resolved.
Cannot both disallow builds and disallow wheels. Please allow one of these or both so that some distributions can be resolved.

With the stacktrace pointing to pex/resolve/lock_resolver.py

class VCSArtifactDownloadManager(DownloadManager[VCSArtifact]):
    def __init__(
        self,
         ...
    ):
        super(VCSArtifactDownloadManager, self).__init__(
            pex_root=pex_root, file_lock_style=file_lock_style
        )
        ...
        self._build_configuration = attr.evolve(
            build_configuration, allow_wheels=False, prefer_older_binary=False
        )
jsirois commented 6 months ago

This was a regression introduced in #2346. It looks like both allow_wheels=False and allow_build=True should be established by the VCSArtifactDownloadManager before taking all other options from the user. And in the case of a --no-build --wheel lock, there will be no VCS artifacts in the lock; thus no VCSArtifactDownloadManager will actually be used to build the PEX and so all is well.

jsirois commented 6 months ago

This is definitely a bug, but I will note a few un-needed things with the OP setup:

  1. --wheel: That's the default (allow wheels and allow builds).
  2. Saying pex --wheel --no-build --lock ... when the lock already does not admit wheels.

In both cases, presumably, this is done either due to liking being explicit, which I don't fault, or some form of automation where its easier to have the pex create options be redundant to the lock create options.

The bug is triggered by using 2 instead of saying nothing, which would have acheived the same ends successfully.