search

pex-tool / pex

A tool for generating .pex (Python EXecutable) files, lock files and venvs.
https://docs.pex-tool.org
Apache License 2.0
2.49k stars 254 forks source link

Support for pip-compile generated requirements with hashes #2412

Closed anisse closed 2 days ago

anisse commented 1 month ago

When attempting to build a pex for package with dependencies pinned with pip-compile --generate-hashes, it will fail with the following error:

ERROR: Can't verify hashes for these file:// requirements because they point to directories:
    file:///home/anisse/dev/minimal-pip-compile-pex

I made a (not so) minimal repro to exhibit this issue, the relevant code can be found here: https://github.com/anisse/pex-minimal-repros/commit/4648ec016a644f91362a5e5984957f03e7053a60

I know there might be a few similar issues, like #429, and that pex has put a lot of effort in its own lock file format, so feel free to close this issue if it's out of scope for the project.

jsirois commented 1 month ago

Pex definitely supports requirements files with hashes - it just passes these through to Pip to handle. I've closed #429 using your repo for the example (the numpy + pip-compile -> "locked" numpy bit). So the issue you mention above in the OP must be very specific to the hash of a project directory. I'm pretty sure that message is coming from Pip but I'll verify.

jsirois commented 1 month ago

Ok, yeah. So here is the full terminal command line from your example:

...
/tmp/tmp5xjc2oji/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /tmp/tmp5xjc2oji/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -v --cache-dir /tmp/tmp5xjc2oji/pip/24.0/pip_cache --log /tmp/pex-pip-log.wdluxqkc/pip.log download --dest /tmp/tmp5xjc2oji/downloads/resolver_download.132v5424/home.jsirois.dev.anisse.pex-minimal-repros..env.bin.python3 --requirement requirements.txt /home/jsirois/dev/anisse/pex-minimal-repros --retries 5 --timeout 15
pid 8909 -> /tmp/tmp5xjc2oji/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /tmp/tmp5xjc2oji/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -v --cache-dir /tmp/tmp5xjc2oji/pip/24.0/pip_cache --log /tmp/pex-pip-log.wdluxqkc/pip.log download --dest /tmp/tmp5xjc2oji/downloads/resolver_download.132v5424/home.jsirois.dev.anisse.pex-minimal-repros..env.bin.python3 --requirement requirements.txt /home/jsirois/dev/anisse/pex-minimal-repros --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Can't verify hashes for these file:// requirements because they point to directories:
    file:///home/jsirois/dev/anisse/pex-minimal-repros

That command line is complex looking, but its actually just Pex executing Pip (itself PEXed up) in a subprocess. The relevant bits are:

(pip) download ... --requirement requirements.txt /home/jsirois/dev/anisse/pex-minimal-repros ...

So bdist_pex is building the current project (. or /home/jsirois/dev/anisse/pex-minimal-repros) and your Makefile adds these args --pex-args='... -r requirements.txt ...'. So yhe full requirement set is. -r requirements.txt. This will always fail since Pip requires every requirement has a hash if any does. The current project directory does not, but the requirements inrequirements.txt` do; so Pip fails the download. In particular, Pip doesn't support hashes for directories (projects) at all anyhow; so this is impossible with Pip. Now Pex does support project directory hashes in its own lock file format, but that doesn't help you here. I can't think of a way to PEX up the combination of a local project directory + hashed requirements. Here's how you'd do this using just Pex though if that's of interest:

# ~/dev/anisse/pex-minimal-repros (4648ec0) [?] via  v3.10.12 took 3s
:; pex3 lock create . -r requirements.in --indent 2 -o lock.json

# ~/dev/anisse/pex-minimal-repros (4648ec0) [?] via  v3.10.12 took 2s
:; cat lock.json
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed",
              "url": "https://files.pythonhosted.org/packages/0f/50/de23fde84e45f5c4fda2488c759b69990fd4512387a8632860f3ac9cd225/numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
            }
          ],
          "project_name": "numpy",
          "requires_dists": [],
          "requires_python": ">=3.9",
          "version": "1.26.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "53c32e4adf05ce880bb57d2e35788de81d3febc9e05aaf474782b81d282b2703",
              "url": "file:///home/jsirois/dev/anisse/pex-minimal-repros"
            }
          ],
          "project_name": "repro",
          "requires_dists": [],
          "requires_python": "~=3.9",
          "version": "0.0.1"
        }
      ],
      "platform_tag": [
        "cp312",
        "cp312",
        "manylinux_2_35_x86_64"
      ]
    }
  ],
  "only_builds": [],
  "only_wheels": [],
  "path_mappings": {},
  "pex_version": "2.3.1",
  "pip_version": "23.2",
  "prefer_older_binary": false,
  "requirements": [
    "numpy",
    "repro==0.0.1"
  ],
  "requires_python": [],
  "resolver_version": "pip-2020-resolver",
  "style": "strict",
  "target_systems": [],
  "transitive": true,
  "use_pep517": null
}

# ~/dev/anisse/pex-minimal-repros (4648ec0) [?] via  v3.10.12
:; pex --lock lock.json -c entry -o dist/entry.pex

# ~/dev/anisse/pex-minimal-repros (4648ec0) [?] via  v3.10.12
:; dist/entry.pex
Hello  1.26.4

Now, that said. I don't think you want this behavior, where the local project itself is part of the lock? Pex, like Pip, requires everything is in a lock when using a lock (in Pip's case, all requirements are hashed when any are). So I think what you're looking for is the ability to build a PEX from a partially locked set of input requirements (just lock 3rdparty requirements; not local projects). That feature does not exist today.

@anisse if you can confirm my guess or clarify what you're aiming for in this example at a high level if my guess was wrong, I'd be grateful.

jsirois commented 1 month ago

FWIW, for apples to apples with the pip-compile you use in your Makefile, you'd likely want this pex3 lock create command line:

# ~/dev/anisse/pex-minimal-repros (4648ec0) [?] via  v3.10.12 took 45s
:; pex3 lock create . -r requirements.in --interpreter-constraint "~=3.9" --style universal --pip-version 24.0 --indent 2 -o lock.json

# ~/dev/anisse/pex-minimal-repros (4648ec0) [?] via  v3.10.12 took 9s
:; cat lock.json
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "7e50d0a0cc3189f9cb0aeb3a6a6af18c16f59f004b866cd2be1c14b36134a4a0",
              "url": "https://files.pythonhosted.org/packages/f4/5f/fafd8c51235f60d49f7a88e2275e13971e90555b67da52dd6416caec32fe/numpy-1.26.4-pp39-pypy39_pp73-win_amd64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "96ff0b2ad353d8f990b63294c8986f1ec3cb19d749234014f4e7eb0112ceba5a",
              "url": "https://files.pythonhosted.org/packages/09/bf/2b1aaf8f525f2923ff6cfcf134ae5e750e279ac65ebf386c75a0cf6da06a/numpy-1.26.4-cp311-cp311-musllinux_1_1_aarch64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed",
              "url": "https://files.pythonhosted.org/packages/0f/50/de23fde84e45f5c4fda2488c759b69990fd4512387a8632860f3ac9cd225/numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "4c66707fabe114439db9068ee468c26bbdf909cac0fb58686a42a24de1760c71",
              "url": "https://files.pythonhosted.org/packages/11/57/baae43d14fe163fa0e4c47f307b6b2511ab8d7d30177c491960504252053/numpy-1.26.4-cp311-cp311-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "08beddf13648eb95f8d867350f6a018a4be2e5ad54c8d8caed89ebca558b2818",
              "url": "https://files.pythonhosted.org/packages/16/2e/86f24451c2d530c88daf997cb8d6ac622c1d40d19f5a031ed68a4b73a374/numpy-1.26.4-cp312-cp312-win_amd64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "47711010ad8555514b434df65f7d7b076bb8261df1ca9bb78f53d3b2db02e95c",
              "url": "https://files.pythonhosted.org/packages/16/ee/9df80b06680aaa23fc6c31211387e0db349e0e36d6a63ba3bd78c5acdf11/numpy-1.26.4-cp39-cp39-musllinux_1_1_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "b97fe8060236edf3662adfc2c633f56a08ae30560c56310562cb4f95500022d5",
              "url": "https://files.pythonhosted.org/packages/19/77/538f202862b9183f54108557bfda67e17603fc560c384559e769321c9d92/numpy-1.26.4-cp310-cp310-win_amd64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "edd8b5fe47dab091176d21bb6de568acdd906d1887a4584a15a9a96a1dca06ef",
              "url": "https://files.pythonhosted.org/packages/1a/2e/151484f49fd03944c4a3ad9c418ed193cfd02724e138ac8a9505d056c582/numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "2e4ee3380d6de9c9ec04745830fd9e2eccb3e6cf790d39d7b98ffd19b0dd754a",
              "url": "https://files.pythonhosted.org/packages/20/f7/b24208eba89f9d1b58c1668bc6c8c4fd472b20c45573cb767f59d49fb0f6/numpy-1.26.4-cp310-cp310-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "62b8e4b1e28009ef2846b4c7852046736bab361f7aeadeb6a5b89ebec3c7055a",
              "url": "https://files.pythonhosted.org/packages/24/03/6f229fe3187546435c4f6f89f6d26c129d4f5bed40552899fcf1f0bf9e50/numpy-1.26.4-cp310-cp310-musllinux_1_1_aarch64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "50193e430acfc1346175fcbdaa28ffec49947a06918b7b92130744e81e640110",
              "url": "https://files.pythonhosted.org/packages/28/4a/46d9e65106879492374999e76eb85f87b15328e06bd1550668f79f7b18c6/numpy-1.26.4-cp312-cp312-win32.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "a354325ee03388678242a4d7ebcd08b5c727033fcff3b2f536aea978e15ee9e6",
              "url": "https://files.pythonhosted.org/packages/28/7d/4b92e2fe20b214ffca36107f1a3e75ef4c488430e64de2d9af5db3a4637d/numpy-1.26.4-cp39-cp39-win32.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "a4abb4f9001ad2858e7ac189089c42178fcce737e4169dc61321660f1a96c7d2",
              "url": "https://files.pythonhosted.org/packages/39/fe/39ada9b094f01f5a35486577c848fe274e374bbf8d8f472e1423a0bbd26d/numpy-1.26.4-cp310-cp310-musllinux_1_1_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "666dbfb6ec68962c033a450943ded891bed2d54e6755e35e5835d63f4f6931d5",
              "url": "https://files.pythonhosted.org/packages/3a/d0/edc009c27b406c4f9cbc79274d6e46d634d139075492ad055e3d68445925/numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "cd25bcecc4974d09257ffcd1f098ee778f7834c3ad767fe5db785be9a4aa9cb2",
              "url": "https://files.pythonhosted.org/packages/3f/6b/5610004206cf7f8e7ad91c5a85a8c71b2f2f8051a0c0c4d5916b76d6cbb2/numpy-1.26.4-cp311-cp311-win_amd64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "afedb719a9dcfc7eaf2287b839d8198e06dcd4cb5d276a3df279231138e83d30",
              "url": "https://files.pythonhosted.org/packages/3f/72/3df6c1c06fc83d9cfe381cccb4be2532bbd38bf93fbc9fad087b6687f1c0/numpy-1.26.4-pp39-pypy39_pp73-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "679b0076f67ecc0138fd2ede3a8fd196dddc2ad3254069bcb9faf9a79b1cebcd",
              "url": "https://files.pythonhosted.org/packages/43/12/01a563fc44c07095996d0129b8899daf89e4742146f7044cdbdb3101c57f/numpy-1.26.4-cp39-cp39-musllinux_1_1_aarch64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "ffa75af20b44f8dba823498024771d5ac50620e6915abac414251bd971b4529f",
              "url": "https://files.pythonhosted.org/packages/4b/d7/ecf66c1cd12dc28b4040b15ab4d17b773b87fa9d29ca16125de01adb36cd/numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "ab47dbe5cc8210f55aa58e4805fe224dac469cde56b9f731a4c098b91917159a",
              "url": "https://files.pythonhosted.org/packages/4c/0c/9c603826b6465e82591e05ca230dfc13376da512b25ccd0894709b054ed0/numpy-1.26.4-cp312-cp312-musllinux_1_1_aarch64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "f870204a840a60da0b12273ef34f7051e98c3b5961b61b0c2c1be6dfd64fbcd3",
              "url": "https://files.pythonhosted.org/packages/54/30/c2a907b9443cf42b90c17ad10c1e8fa801975f01cb9764f3f8eb8aea638b/numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010",
              "url": "https://files.pythonhosted.org/packages/65/6e/09db70a523a96d25e115e71cc56a6f9031e7b8cd166c1ac8438307c14058/numpy-1.26.4.tar.gz"
            },
            {
              "algorithm": "sha256",
              "hash": "d5241e0a80d808d70546c697135da2c613f30e28251ff8307eb72ba696945764",
              "url": "https://files.pythonhosted.org/packages/6d/64/c3bcdf822269421d85fe0d64ba972003f9bb4aa9a419da64b86856c9961f/numpy-1.26.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "03a8c78d01d9781b28a6989f6fa1bb2c4f2d51201cf99d3dd875df6fbd96b23b",
              "url": "https://files.pythonhosted.org/packages/75/5b/ca6c8bd14007e5ca171c7c03102d17b4f4e0ceb53957e8c44343a9546dcc/numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "1dda2e7b4ec9dd512f84935c5f126c8bd8b9f2fc001e9f54af255e8c5f16b0e0",
              "url": "https://files.pythonhosted.org/packages/76/8c/2ba3902e1a0fc1c74962ea9bb33a534bb05984ad7ff9515bf8d07527cadd/numpy-1.26.4-cp312-cp312-musllinux_1_1_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "7ab55401287bfec946ced39700c053796e7cc0e3acbef09993a9ad2adba6ca6e",
              "url": "https://files.pythonhosted.org/packages/79/ae/7e5b85136806f9dadf4878bf73cf223fe5c2636818ba3ab1c585d0403164/numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "9fad7dcb1aac3c7f0584a5a8133e3a43eeb2fe127f47e3632d43d677c66c102b",
              "url": "https://files.pythonhosted.org/packages/79/f8/97f10e6755e2a7d027ca783f63044d5b1bc1ae7acb12afe6a9b4286eac17/numpy-1.26.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "7349ab0fa0c429c82442a27a9673fc802ffdb7c7775fad780226cb234965e53c",
              "url": "https://files.pythonhosted.org/packages/7d/24/ce71dc08f06534269f66e73c04f5709ee024a1afe92a7b6e1d73f158e1f8/numpy-1.26.4-cp39-cp39-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "95a7476c59002f2f6c590b9b7b998306fba6a5aa646b1e22ddfeaf8f78c3a29c",
              "url": "https://files.pythonhosted.org/packages/8e/02/570545bac308b58ffb21adda0f4e220ba716fb658a63c151daecc3293350/numpy-1.26.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "b3ce300f3644fb06443ee2222c2201dd3a89ea6040541412b8fa189341847218",
              "url": "https://files.pythonhosted.org/packages/95/12/8f2020a8e8b8383ac0177dc9570aad031a3beb12e38847f7129bacd96228/numpy-1.26.4-cp312-cp312-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "9ff0f4f29c51e2803569d7a51c2304de5554655a60c5d776e35b4a41413830d0",
              "url": "https://files.pythonhosted.org/packages/a7/94/ace0fdea5241a27d13543ee117cbc65868e82213fb31a8eb7fe9ff23f313/numpy-1.26.4-cp310-cp310-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "52b8b60467cd7dd1e9ed082188b4e6bb35aa5cdd01777621a1658910745b90be",
              "url": "https://files.pythonhosted.org/packages/ae/8c/ab03a7c25741f9ebc92684a20125fbc9fc1b8e1e700beb9197d750fdff88/numpy-1.26.4-cp39-cp39-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "3373d5d70a5fe74a2c1bb6d2cfd9609ecf686d47a2d7b1d37a8f3b6bf6003aea",
              "url": "https://files.pythonhosted.org/packages/b5/42/054082bd8220bbf6f297f982f0a8f5479fcbc55c8b511d928df07b965869/numpy-1.26.4-cp39-cp39-win_amd64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "1af303d6b2210eb850fcf03064d364652b7120803a0b872f5211f5234b399f20",
              "url": "https://files.pythonhosted.org/packages/d2/b7/a734c733286e10a7f1a8ad1ae8c90f2d33bf604a96548e0a4a3a6739b468/numpy-1.26.4-cp311-cp311-win32.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "bfe25acf8b437eb2a8b2d49d443800a5f18508cd811fea3181723922a8a82b07",
              "url": "https://files.pythonhosted.org/packages/d5/ef/6ad11d51197aad206a9ad2286dc1aac6a378059e06e8cf22cd08ed4f20dc/numpy-1.26.4-cp310-cp310-win32.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "60dedbb91afcbfdc9bc0b1f3f402804070deed7392c23eb7a7f07fa857868e8a",
              "url": "https://files.pythonhosted.org/packages/df/a0/4e0f14d847cfc2a633a1c8621d00724f3206cfeddeb66d35698c4e2cf3d2/numpy-1.26.4-cp311-cp311-musllinux_1_1_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "d209d8969599b27ad20994c8e41936ee0964e6da07478d6c35016bc386b66ad4",
              "url": "https://files.pythonhosted.org/packages/fc/a5/4beee6488160798683eed5bdb7eead455892c3b4e1f78d79d8d3f3b084ac/numpy-1.26.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
            }
          ],
          "project_name": "numpy",
          "requires_dists": [],
          "requires_python": ">=3.9",
          "version": "1.26.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "53c32e4adf05ce880bb57d2e35788de81d3febc9e05aaf474782b81d282b2703",
              "url": "file:///home/jsirois/dev/anisse/pex-minimal-repros"
            }
          ],
          "project_name": "repro",
          "requires_dists": [],
          "requires_python": "~=3.9",
          "version": "0.0.1"
        }
      ],
      "platform_tag": null
    }
  ],
  "only_builds": [],
  "only_wheels": [],
  "path_mappings": {},
  "pex_version": "2.3.1",
  "pip_version": "24.0",
  "prefer_older_binary": false,
  "requirements": [
    "numpy",
    "repro==0.0.1"
  ],
  "requires_python": [
    "~=3.9"
  ],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "target_systems": [],
  "transitive": true,
  "use_pep517": null
}

# ~/dev/anisse/pex-minimal-repros (4648ec0) [?] via  v3.10.12
:; pex --lock lock.json -c entry -o dist/entry.pex

# ~/dev/anisse/pex-minimal-repros (4648ec0) [?] via  v3.10.12
:; dist/entry.pex
Hello  1.26.4

Unlike pip-compile - and most Python lock file systems I know of today, Pex defaults to a strict narrow lock that just locks exactly the artifacts needed by the Python(s) targeted by the pex3 lock create ... command line. As such each locked requirement has one and only one hashed artifact - the one that was locked (and presumably actually tested) using the current targeted interpreter. The --interpreter-constraint ~=3.9 --style universal tells pex3 lock create to relax the lock and include artifacts for all Pythons implied by ~=3.9. This will include sdists and other un-vetted wheels not used (or tested) under the current interpreter. That's what pip-compile does by default. I was uncomfortable with that less-than-secure-by-default mode (I think its the only mode for pip-compile~, Poetry, PDM, etc); so I created pex3 lock create to be conservative by default on day 1.

anisse commented 1 month ago

So I think what you're looking for is the ability to build a PEX from a partially locked set of input requirements (just lock 3rdparty requirements; not local projects). That feature does not exist today.

Yes, that's exactly that, a bit like what lock files do in the js, go or rust ecosystem. Note that in a venv, one can get close to this result, by first doing a pip install -r <requirements-with-hashes>, followed by a second pip install .. I realize this might not be a pex issue, and just the way pip behaves since pex invokes "pip_download". Wouldn't be possible to invoke it twice to reach a similar result?

I wanted to investigate using pex locks, which might match my requirements, but then it meant having at least two sets of lockfiles (one for the generated requirements.txt, one for each python version for which we want to generate a pex); at that point the cons might outweigh the gains, and just having "release" generated pex files properly archived and traced might be enough (without a lockfile).

jsirois commented 6 days ago

Ok, finally looking back at this. I think what this means then using examples is:

  1. Lock 3rdparty for all supported Pythons (~=3.9) and all platforms (Linux and Mac): 
    pex3 lock create -r requirements.in --target-system linux --target-system mac --interpreter-constraint "~=3.9" --style universal --pip-version 24.0 --indent 2 -o lock.json
  2. Build local project PEX from lock: 
    pex --project . --lock lock.json -c entry -o dist/entry.pex

The new feature would be --project . in step 2. Instead of using a bare ., which today means a local project requirement to be looked up in the lock, the --project . would mean to 1st build the local project at . and then resolve all its dependencies from the --lock (and fail if it had dependencies not in the --lock).

With this, step 1 could also be done like so:

pex3 lock create --project . --target-system linux --target-system mac --interpreter-constraint "~=3.9" --style universal --pip-version 24.0 --indent 2 -o lock.json

Here the -r requirements.in is replaced with --project .. The meaning is the same as in the step 2 example: 1st build the --project at ., and then lock its requirements.

jsirois commented 3 days ago

Ok, missed a small change in #2455 to get bdist_pex actually using --project. I'll patch that up and add an IT ~identical to your OP repro repo commit to make sure this is / stays fixed this time.

jsirois commented 3 days ago

I wanted to investigate using pex locks, which might match my requirements, but then it meant having at least two sets of lockfiles (one for the generated requirements.txt, one for each python version for which we want to generate a pex); at that point the cons might outweigh the gains, and just having "release" generated pex files properly archived and traced might be enough (without a lockfile).

I wanted to circle back and highlight that I don't think this is true. A pex3 lock {create,sync} --style universal --interpreter-constraint ~=3.9 ... lock will lock for all python versions implied by the interpreter constraint as well as all platforms; so you only need 1 lock file. If you need a hashed requirements file for interoperability with other tools, those are per-interpreter, but don't really need to be checked in since they can be derived reproducibly (and quickly) from the checked in lock file with:

some-other-tool-in-activated-venv-that-also-has-pex-installed -r <(pex3 lock export lock.json)

For example:

# Given a universal lock.
:; pex3 lock create numpy --style universal --interpreter-constraint ~=3.9 --pip-version latest --indent 2 -o lock.json

:; python3.11 -mvenv 311.venv
311.venv/bin/pip -q install -U pip pex
311.venv/bin/pip list
Package    Version
---------- -------
pex        2.8.0
pip        24.1.1
setuptools 65.5.0
:; 311.venv/bin/pip -q install --require-hashes -r <(311.venv/bin/pex3 lock export lock.json)
:; 311.venv/bin/pip list
Package    Version
---------- -------
numpy      2.0.0
pex        2.8.0
pip        24.1.1
setuptools 65.5.0

# Note that the ephemeral hashed requirements used are exactly for the subset of applicable dists usable by Python 3.11 (1 wheel hash, 1 sdist hash):
:; time 311.venv/bin/pex3 lock export lock.json
numpy==2.0.0 \
  --hash=sha256:a7039a136017eaa92c1848152827e1424701532ca8e8967fe480fe1569dae581 \
  --hash=sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864

real    0m0.169s
user    0m0.169s
sys     0m0.000s

# And that set changes for different Pythons since the wheel is different (or there is no wheel at all):
:; time for v in `seq 9 13`; do which python3.$v && 311.venv/bin/pex3 lock export --python python3.$v lock.json && echo; done
/home/jsirois/.pyenv/shims/python3.9
numpy==2.0.0 \
  --hash=sha256:821eedb7165ead9eebdb569986968b541f9908979c2da8a4967ecac4439bae3d \
  --hash=sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864

/home/jsirois/.pyenv/shims/python3.10
numpy==2.0.0 \
  --hash=sha256:6d7696c615765091cc5093f76fd1fa069870304beaccfd58b5dcc69e55ef49c1 \
  --hash=sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864

/home/jsirois/.pyenv/shims/python3.11
numpy==2.0.0 \
  --hash=sha256:a7039a136017eaa92c1848152827e1424701532ca8e8967fe480fe1569dae581 \
  --hash=sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864

/home/jsirois/.pyenv/shims/python3.12
numpy==2.0.0 \
  --hash=sha256:49d9f7d256fbc804391a7f72d4a617302b1afac1112fac19b6c6cec63fe7fe8a \
  --hash=sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864

/home/jsirois/.pyenv/shims/python3.13
numpy==2.0.0 \
  --hash=sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864

real    0m0.847s
user    0m0.733s
sys     0m0.091s

# Of course all these hashes are there in the lock file - there is no network access or resolving when exporting:
:; jq -r '.locked_resolves[] | .locked_requirements[] | .artifacts[] | .algorithm + ":" + .hash + " " + (.url | sub("^https.*/"; ""))' lock.json
sha256:cef04d068f5fb0518a77857953193b6bb94809a806bd0a14983a8f12ada060c9 numpy-2.0.0-pp39-pypy39_pp73-win_amd64.whl
sha256:feff59f27338135776f6d4e2ec7aeeac5d5f7a08a83e80869121ef8164b74af9 numpy-2.0.0-cp312-cp312-musllinux_1_2_aarch64.whl
sha256:34f003cb88b1ba38cb9a9a4a3161c1604973d7f9d5552c38bc2f04f829536609 numpy-2.0.0-cp311-cp311-macosx_14_0_arm64.whl
sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864 numpy-2.0.0.tar.gz
sha256:17067d097ed036636fa79f6a869ac26df7db1ba22039d962422506640314933a numpy-2.0.0-pp39-pypy39_pp73-macosx_14_0_x86_64.whl
sha256:46e161722e0f619749d1cd892167039015b2c2817296104487cd03ed4a955995 numpy-2.0.0-cp311-cp311-musllinux_1_1_x86_64.whl
sha256:acd3a644e4807e73b4e1867b769fbf1ce8c5d80e7caaef0d90dcdc640dfc9787 numpy-2.0.0-cp310-cp310-musllinux_1_2_aarch64.whl
sha256:c5a59996dc61835133b56a32ebe4ef3740ea5bc19b3983ac60cc32be5a665d54 numpy-2.0.0-cp312-cp312-win32.whl
sha256:49d9f7d256fbc804391a7f72d4a617302b1afac1112fac19b6c6cec63fe7fe8a numpy-2.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sha256:04494f6ec467ccb5369d1808570ae55f6ed9b5809d7f035059000a37b8d7e86f numpy-2.0.0-cp310-cp310-macosx_10_9_x86_64.whl
sha256:0a43f0974d501842866cc83471bdb0116ba0dffdbaac33ec05e6afed5b615238 numpy-2.0.0-cp310-cp310-macosx_14_0_arm64.whl
sha256:1cde1753efe513705a0c6d28f5884e22bdc30438bf0085c5c486cdaff40cd67a numpy-2.0.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
sha256:1e72728e7501a450288fc8e1f9ebc73d90cfd4671ebbd631f3e7857c39bd16f2 numpy-2.0.0-cp312-cp312-macosx_14_0_arm64.whl
sha256:b6f6a8f45d0313db07d6d1d37bd0b112f887e1369758a5419c0370ba915b3871 numpy-2.0.0-cp311-cp311-macosx_14_0_x86_64.whl
sha256:63b92c512d9dbcc37f9d81b123dec99fdb318ba38c8059afc78086fe73820275 numpy-2.0.0-cp39-cp39-win32.whl
sha256:b4c76e3d4c56f145d41b7b6751255feefae92edbc9a61e1758a98204200f30fc numpy-2.0.0-cp310-cp310-musllinux_1_1_x86_64.whl
sha256:ad0c86f3455fbd0de6c31a3056eb822fc939f81b1618f10ff3406971893b62a5 numpy-2.0.0-cp311-cp311-macosx_10_9_x86_64.whl
sha256:3f6bed7f840d44c08ebdb73b1825282b801799e325bcbdfa6bc5c370e5aecc65 numpy-2.0.0-cp39-cp39-win_amd64.whl
sha256:e7f387600d424f91576af20518334df3d97bc76a300a755f9a8d6e4f5cadd289 numpy-2.0.0-cp311-cp311-macosx_11_0_arm64.whl
sha256:e61155fae27570692ad1d327e81c6cf27d535a5d7ef97648a17d922224b216de numpy-2.0.0-cp39-cp39-macosx_10_9_x86_64.whl
sha256:38ecb5b0582cd125f67a629072fed6f83562d9dd04d7e03256c9829bdec027ad numpy-2.0.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sha256:9c27f0946a3536403efb0e1c28def1ae6730a72cd0d5878db38824855e3afc44 numpy-2.0.0-cp39-cp39-musllinux_1_2_aarch64.whl
sha256:821eedb7165ead9eebdb569986968b541f9908979c2da8a4967ecac4439bae3d numpy-2.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sha256:cee6cc0584f71adefe2c908856ccc98702baf95ff80092e4ca46061538a2ba98 numpy-2.0.0-cp310-cp310-win32.whl
sha256:903703372d46bce88b6920a0cd86c3ad82dae2dbef157b5fc01b70ea1cfc430f numpy-2.0.0-cp39-cp39-macosx_14_0_arm64.whl
sha256:9416a5c2e92ace094e9f0082c5fd473502c91651fb896bc17690d6fc475128d6 numpy-2.0.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl
sha256:fbd6acc766814ea6443628f4e6751d0da6593dae29c08c0b2606164db026970c numpy-2.0.0-cp311-cp311-win_amd64.whl
sha256:ed08d2703b5972ec736451b818c2eb9da80d66c3e84aed1deeb0c345fefe461b numpy-2.0.0-cp310-cp310-win_amd64.whl
sha256:84554fc53daa8f6abf8e8a66e076aff6ece62de68523d9f665f32d2fc50fd66e numpy-2.0.0-cp312-cp312-macosx_14_0_x86_64.whl
sha256:a356364941fb0593bb899a1076b92dfa2029f6f5b8ba88a14fd0984aaf76d0df numpy-2.0.0-cp312-cp312-win_amd64.whl
sha256:2635dbd200c2d6faf2ef9a0d04f0ecc6b13b3cad54f7c67c61155138835515d2 numpy-2.0.0-cp310-cp310-macosx_11_0_arm64.whl
sha256:354f373279768fa5a584bac997de6a6c9bc535c482592d7a813bb0c09be6c76f numpy-2.0.0-cp312-cp312-macosx_10_9_x86_64.whl
sha256:c73aafd1afca80afecb22718f8700b40ac7cab927b8abab3c3e337d70e10e5a2 numpy-2.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
sha256:4554eb96f0fd263041baf16cf0881b3f5dafae7a59b1049acb9540c4d57bc8cb numpy-2.0.0-cp39-cp39-macosx_11_0_arm64.whl
sha256:9a1712c015831da583b21c5bfe15e8684137097969c6d22e8316ba66b5baabe4 numpy-2.0.0-cp39-cp39-musllinux_1_1_x86_64.whl
sha256:0e50842b2295ba8414c8c1d9d957083d5dfe9e16828b37de883f51fc53c4016f numpy-2.0.0-cp311-cp311-musllinux_1_2_aarch64.whl
sha256:5f64641b42b2429f56ee08b4f427a4d2daf916ec59686061de751a55aafa22e4 numpy-2.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
sha256:a7039a136017eaa92c1848152827e1424701532ca8e8967fe480fe1569dae581 numpy-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sha256:3e8e01233d57639b2e30966c63d36fcea099d17c53bf424d77f088b0f4babd86 numpy-2.0.0-cp39-cp39-macosx_14_0_x86_64.whl
sha256:6d7696c615765091cc5093f76fd1fa069870304beaccfd58b5dcc69e55ef49c1 numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sha256:0ec84b9ba0654f3b962802edc91424331f423dcf5d5f926676e0150789cb3d95 numpy-2.0.0-cp312-cp312-musllinux_1_1_x86_64.whl
sha256:79e843d186c8fb1b102bef3e2bc35ef81160ffef3194646a7fdd6a73c6b97196 numpy-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
sha256:2ce46fd0b8a0c947ae047d222f7136fc4d55538741373107574271bc00e20e8f numpy-2.0.0-cp311-cp311-win32.whl
sha256:8d83bb187fb647643bd56e1ae43f273c7f4dbcdf94550d7938cfc32566756514 numpy-2.0.0-cp310-cp310-macosx_14_0_x86_64.whl
sha256:4d2f62e55a4cd9c58c1d9a1c9edaedcd857a73cb6fda875bf79093f9d9086f85 numpy-2.0.0-cp312-cp312-macosx_11_0_arm64.whl
jsirois commented 2 days ago

Alright @anisse, with Pex 2.8.1 released your example at https://github.com/anisse/pex-minimal-repros/commit/4648ec016a644f91362a5e5984957f03e7053a60 works as-is:

:; git reset --hard 4648ec016a644f91362a5e5984957f03e7053a60
HEAD is now at 4648ec0 initial version reproducing the pex + hashes issue

# ~/dev/anisse/pex-minimal-repros (4648ec0) [$] via  v3.10.12 on ☁️
:; make clean
rm -fr ".env" dist requirements.txt __pycache__

# ~/dev/anisse/pex-minimal-repros (4648ec0) [$] via  v3.10.12 on ☁️
:; make
python3 -m venv .env
./.env/bin/pip install --upgrade pip-tools pip pex
Collecting pip-tools
  Using cached pip_tools-7.4.1-py3-none-any.whl (61 kB)
Requirement already satisfied: pip in ./.env/lib/python3.10/site-packages (22.0.2)
Collecting pip
  Using cached pip-24.1.2-py3-none-any.whl (1.8 MB)
Collecting pex
  Downloading pex-2.8.1-py2.py3-none-any.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 8.3 MB/s eta 0:00:00
Collecting wheel
  Using cached wheel-0.43.0-py3-none-any.whl (65 kB)
Collecting pyproject-hooks
  Using cached pyproject_hooks-1.1.0-py3-none-any.whl (9.2 kB)
Collecting build>=1.0.0
  Using cached build-1.2.1-py3-none-any.whl (21 kB)
Collecting click>=8
  Using cached click-8.1.7-py3-none-any.whl (97 kB)
Collecting tomli
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from pip-tools) (59.6.0)
Collecting packaging>=19.1
  Using cached packaging-24.1-py3-none-any.whl (53 kB)
Installing collected packages: wheel, tomli, pyproject-hooks, pip, pex, packaging, click, build, pip-tools
  Attempting uninstall: pip
    Found existing installation: pip 22.0.2
    Uninstalling pip-22.0.2:
      Successfully uninstalled pip-22.0.2
Successfully installed build-1.2.1 click-8.1.7 packaging-24.1 pex-2.8.1 pip-24.1.2 pip-tools-7.4.1 pyproject-hooks-1.1.0 tomli-2.0.1 wheel-0.43.0
./.env/bin/pip-compile --output-file=requirements.txt --generate-hashes requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --generate-hashes --output-file=requirements.txt requirements.in
#
numpy==2.0.0 \
    --hash=sha256:04494f6ec467ccb5369d1808570ae55f6ed9b5809d7f035059000a37b8d7e86f \
    --hash=sha256:0a43f0974d501842866cc83471bdb0116ba0dffdbaac33ec05e6afed5b615238 \
    --hash=sha256:0e50842b2295ba8414c8c1d9d957083d5dfe9e16828b37de883f51fc53c4016f \
    --hash=sha256:0ec84b9ba0654f3b962802edc91424331f423dcf5d5f926676e0150789cb3d95 \
    --hash=sha256:17067d097ed036636fa79f6a869ac26df7db1ba22039d962422506640314933a \
    --hash=sha256:1cde1753efe513705a0c6d28f5884e22bdc30438bf0085c5c486cdaff40cd67a \
    --hash=sha256:1e72728e7501a450288fc8e1f9ebc73d90cfd4671ebbd631f3e7857c39bd16f2 \
    --hash=sha256:2635dbd200c2d6faf2ef9a0d04f0ecc6b13b3cad54f7c67c61155138835515d2 \
    --hash=sha256:2ce46fd0b8a0c947ae047d222f7136fc4d55538741373107574271bc00e20e8f \
    --hash=sha256:34f003cb88b1ba38cb9a9a4a3161c1604973d7f9d5552c38bc2f04f829536609 \
    --hash=sha256:354f373279768fa5a584bac997de6a6c9bc535c482592d7a813bb0c09be6c76f \
    --hash=sha256:38ecb5b0582cd125f67a629072fed6f83562d9dd04d7e03256c9829bdec027ad \
    --hash=sha256:3e8e01233d57639b2e30966c63d36fcea099d17c53bf424d77f088b0f4babd86 \
    --hash=sha256:3f6bed7f840d44c08ebdb73b1825282b801799e325bcbdfa6bc5c370e5aecc65 \
    --hash=sha256:4554eb96f0fd263041baf16cf0881b3f5dafae7a59b1049acb9540c4d57bc8cb \
    --hash=sha256:46e161722e0f619749d1cd892167039015b2c2817296104487cd03ed4a955995 \
    --hash=sha256:49d9f7d256fbc804391a7f72d4a617302b1afac1112fac19b6c6cec63fe7fe8a \
    --hash=sha256:4d2f62e55a4cd9c58c1d9a1c9edaedcd857a73cb6fda875bf79093f9d9086f85 \
    --hash=sha256:5f64641b42b2429f56ee08b4f427a4d2daf916ec59686061de751a55aafa22e4 \
    --hash=sha256:63b92c512d9dbcc37f9d81b123dec99fdb318ba38c8059afc78086fe73820275 \
    --hash=sha256:6d7696c615765091cc5093f76fd1fa069870304beaccfd58b5dcc69e55ef49c1 \
    --hash=sha256:79e843d186c8fb1b102bef3e2bc35ef81160ffef3194646a7fdd6a73c6b97196 \
    --hash=sha256:821eedb7165ead9eebdb569986968b541f9908979c2da8a4967ecac4439bae3d \
    --hash=sha256:84554fc53daa8f6abf8e8a66e076aff6ece62de68523d9f665f32d2fc50fd66e \
    --hash=sha256:8d83bb187fb647643bd56e1ae43f273c7f4dbcdf94550d7938cfc32566756514 \
    --hash=sha256:903703372d46bce88b6920a0cd86c3ad82dae2dbef157b5fc01b70ea1cfc430f \
    --hash=sha256:9416a5c2e92ace094e9f0082c5fd473502c91651fb896bc17690d6fc475128d6 \
    --hash=sha256:9a1712c015831da583b21c5bfe15e8684137097969c6d22e8316ba66b5baabe4 \
    --hash=sha256:9c27f0946a3536403efb0e1c28def1ae6730a72cd0d5878db38824855e3afc44 \
    --hash=sha256:a356364941fb0593bb899a1076b92dfa2029f6f5b8ba88a14fd0984aaf76d0df \
    --hash=sha256:a7039a136017eaa92c1848152827e1424701532ca8e8967fe480fe1569dae581 \
    --hash=sha256:acd3a644e4807e73b4e1867b769fbf1ce8c5d80e7caaef0d90dcdc640dfc9787 \
    --hash=sha256:ad0c86f3455fbd0de6c31a3056eb822fc939f81b1618f10ff3406971893b62a5 \
    --hash=sha256:b4c76e3d4c56f145d41b7b6751255feefae92edbc9a61e1758a98204200f30fc \
    --hash=sha256:b6f6a8f45d0313db07d6d1d37bd0b112f887e1369758a5419c0370ba915b3871 \
    --hash=sha256:c5a59996dc61835133b56a32ebe4ef3740ea5bc19b3983ac60cc32be5a665d54 \
    --hash=sha256:c73aafd1afca80afecb22718f8700b40ac7cab927b8abab3c3e337d70e10e5a2 \
    --hash=sha256:cee6cc0584f71adefe2c908856ccc98702baf95ff80092e4ca46061538a2ba98 \
    --hash=sha256:cef04d068f5fb0518a77857953193b6bb94809a806bd0a14983a8f12ada060c9 \
    --hash=sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864 \
    --hash=sha256:e61155fae27570692ad1d327e81c6cf27d535a5d7ef97648a17d922224b216de \
    --hash=sha256:e7f387600d424f91576af20518334df3d97bc76a300a755f9a8d6e4f5cadd289 \
    --hash=sha256:ed08d2703b5972ec736451b818c2eb9da80d66c3e84aed1deeb0c345fefe461b \
    --hash=sha256:fbd6acc766814ea6443628f4e6751d0da6593dae29c08c0b2606164db026970c \
    --hash=sha256:feff59f27338135776f6d4e2ec7aeeac5d5f7a08a83e80869121ef8164b74af9
    # via -r requirements.in
# this works:
#./.env/bin/pip-compile --output-file=requirements.txt requirements.in
./.env/bin/python3 setup.py bdist_pex --pex-args='--disable-cache -vvvv -r requirements.txt --pip-version 24.0' --bdist-all
running bdist_pex
Writing entry to /home/jsirois/dev/anisse/pex-minimal-repros/dist/entry

# ~/dev/anisse/pex-minimal-repros (4648ec0) [$?] via  v3.10.12 on ☁️   took 17s
:; dist/entry
Hello  2.0.0

Thanks for bringing this to my attention. The --project feature was sorely missing.