pex-tool / pex

A tool for generating .pex (Python EXecutable) files, lock files and venvs.
https://docs.pex-tool.org/
Apache License 2.0
2.53k stars 259 forks source link

lock update does not preserve "binary-ness" and errors out with artifact set changes #2342

Closed cburroughs closed 6 months ago

cburroughs commented 8 months ago
$ pex3 --version
2.1.160
$ cat __pip_args.txt 
--only-binary acryl-datahub
$ pex3 lock create '--indent=2'  --manylinux manylinux2014 --interpreter-constraint $'CPython==3.10.*' --style=universal --pip-version=23.3.2 --resolver-version pip-2020-resolver --target-system=linux --target-system=mac  -r __pip_args.txt acryl-datahub==0.12.0.5 ansicolors~=1.1.8 -o only.lock
$ pex3 lock update -p ansicolors only.lock 
Encountered 1 error updating only.lock:
1.) cp311-cp311-manylinux_2_37_x86_64: The locked requirement acryl-datahub==0.12.0.5 should have been undisturbed by the lock update, but its additional artifact set changed from:

to:
('https://files.pythonhosted.org/packages/93/9a/bb6a21624adf4bf81534c573a84976b4b17ac21bccae6b00f7e46ed8c1aa/acryl-datahub-0.12.0.5.tar.gz', Fingerprint(algorithm='sha256', hash='6b681b9a735450c0457957a66e1e2b075e662c1fee3fbb4f3a20a192215cd8b9'))
jsirois commented 8 months ago

@cburroughs I think you knew this couldn't work? As per the discussion in https://github.com/pantsbuild/pants/pull/20364#discussion_r1451612149 the --only-binary is a Pip back door Pants is using that Pex knows nothing about; so Pex would need to become globally aware of / support --only-binary first class for this to work at all. That support would be general, and locks would only happen to get support as well.

That said, pex3 lock sync (see: https://github.com/pantsbuild/pex/issues/2334#issuecomment-1900963246) will support the backdoor since it will accept requirement files and that may be what you want to use anyhow.

jsirois commented 8 months ago

So, @cburroughs if you actually want 1st class --no-binary / --only-binary: https://github.com/pantsbuild/pex/issues/2343 If you instead actually just want pex3 lock sync: https://github.com/pantsbuild/pex/issues/2344

It would be good to comment on what you actually want so I can direct current effort. As it stands, I've been chipping away at pex3 lock sync (#2344).

cburroughs commented 8 months ago

I think you knew this couldn't work?

I knew there wasn't any mechanism to change --only/no-binary with update, but I hadn't realized the implication that it would prevent updateing lockfiles created with those settings.

If pex3 lock sync existed I would use that instead of the "sharp knife". (Both as Pex user and for the thinnest plumbing through Pants.)

jsirois commented 8 months ago

@cburroughs great, thanks. I'll continue as-is then and complete #2344 1st then. I will note that Pants current use of a lock file (invalid JSON) header, could / should be eliminated [^1] when switching to pex3 lock sync since that command will noop if that's the right thing to do.

[^1]: It should have been eliminated long ago regardless of pex3 lock sync availability (https://github.com/pantsbuild/pants/issues/14281), but this perhaps makes it easier since, afact, there is no longer a need to store that data anywhere at all.

jsirois commented 7 months ago

Ok, it's been too long. @cburroughs I interrupted myself with activities related to extracting Pex from https://github.com/pantsbuild and https://pantsbuild.org to https://github.com/pex-tool and https://docs.pex-tool.org but that's all wrapped up. I'm back on #2344 and should have that out for review by EOD. I'll invite you to that review - your feedback will be important to help ensure your needs will all be addressed there.

jsirois commented 6 months ago

This issue should now be addressed by #2343 and #2344, the latter of which is now released in Pex 2.3.0: https://github.com/pex-tool/pex/releases/tag/v2.3.0