pex lock update: mechanism to change the version specifier if an existing requirement (or constraint?)

[cburroughs]

With --project= one can adjust the current version of a project anywhere within the existing version specifier, but not change the specifier without regenerating the entire lockfile. This would be useful for the same reasons as adding a new project with the lock is.

I'm unsure if constraints need separate consideration for changing the version specifier. That is pip let's you mange them separately:

$ cat c.txt 
ansicolors<1.1.7
$ pip install -c c.txt  ansicolors==1.0.2

But I'm not sure if that means they must map to separate cli flags in Pex.

[jsirois]

I'm unsure if constraints need separate consideration for changing the version specifier. That is pip let's you mange them separately:

If you read a lock file you'll see original requirements and original constraints are both stored separately (and passed to Pip separately, thus motivating this). So, as you point out, there will be noodling required.

[jsirois]

Ok, I've got -p =<req> (The = prefix is the signal) working for changing an existing lock requirement. Combined with #2332 you could then perform a minimal update for a given lock and a current given set of requirements by:

1. Calculate which locked requirement projects are not in the current given set of requirements: form a -d <project> arg for each of these.
2. For each of the current given set of requirements, form a -p =<req> argument.

That's pretty damn clunky for, presumably, a common use case: auto-update my lock to match current requirements, but with minimal churn. If pex3 lock update were to gain the standard set of requirement options other Pex commands accept, this case could be handled cleanly.

The case breakdown would then be the following 3 mutually exclusive cases: 1.) [existing] A --pin update: no -p, -d or requirements are allowed -> perform a pinned migration. 2.) [existing] A targeted update: -p and -d are used to pinpoint updates to make. No --pin or requirements are allowed. You can pass -r, but when the requirements file is parsed it should contain 0 requirements; just Pip directives. 3.) [new] A minimal sync to new requirements: requirements are passed using requirement strings and -r as in other Pex commands and neither --pin nor -p / -d updates are allowed. All requirements are parsed and treated as -p =<req> requirements and any existing requirements written down in the lock that don't have overlap are treated as -d <proj>.

I think this is backwards compatible, sane, and covers all the use cases that would seem to be typical. The hierarchy is that 2 is the fundamental mode and the sharp knife. 1 is already built on 2 and 3 would now be as well. So both 1 and 3 can be done with 2 but they save you alot of typing as well as saving you from having to know about lock contents.

[jsirois]

Re https://github.com/pantsbuild/pex/issues/2334#issuecomment-1897604016 above, 2 notions lead me to think breaking out this auto-minimal-update support to a new pex3 lock subcommand makes sense:

1. The -R / --replace-project, -r requirements.txt collision on the [rR] switch. [^1]
2. It would be nice to be able to compose a lock sync with running a thing in the results of that:

   pex3 lock sync -r reqs1.txt -r reqs2.txt --lock lock.json -- pytest ...

Item 2 would be nice to have over in science, where I'm almost using pex as my build system today, but I have to do a good bit of plumbing in the outer noxfile.py. The idea is to perform a minimal lock update (or noop if reqs have not changed) and, if -- is present, execute the command line following it after syncing that command's venv with the lock. There is some logic that would be needed to find the venv enclosing argv0 or fail fast on the venv sync part.

So, even without implementing the -- portion of pex3 lock sync right away, this new notion of an ~auto minimal lock update would still use the new features from #2332 and this issue; but it would just get a new CLI sub-command name. That seems reasonable since I expect that with this feature, it's what most folks would use to update locks. In fact sync could stand in for create as well since it accepts all the same requirements args and it could just see that a missing specified lock file meant to create instead of update.

In Git parlance, pex3 lock create and pex3 lock update would be demoted in practice to plumbing and pex3 lock sync would become the porcelain.

[^1]: I still think breaking out pex3 lock sync makes sense, but keeping -R for pex3 lock update probably makes sense. If you use a tool that has switches change meaning based on sub-command, that's probably a bit too confusing.

[jsirois]

This is now implemented in https://github.com/pantsbuild/pex/releases/tag/v2.1.160

[cburroughs]

Thank you for implementing these features! I was called away on personal issues for most of the past week but will attempt to give 2.1.160 a thorough testing this week.