Python: consider adding support for local projects (not local requirements)

[Eric-Arellano]

We now robustly support local requirements, i.e. having .whl and sdist files on the filesystem. But we don't yet support local projects, like my/project, unlike Pex: https://github.com/pantsbuild/pants/pull/16584#discussion_r953894754. This is a useful feature for debugging because it avoids needing to first package the local project.

[jsirois]

I sure hope this is s/consider/do/! It was my strong impression local project locking was required of Pex by Pants. If not, I'd like to know. There is alot of code I'd be happy to kill in Pex that supports this, including a PEP-517 / PEP-518 sdist builder that was needed to get this working.

[jsirois]

See here: https://github.com/pantsbuild/pex/issues/1696#issuecomment-1086661033

[Eric-Arellano]

Oof, that was some serious misunderstanding on my part between local requirements vs projects. I believe the thing Pants users cared the most about is local requirements, although projects would be nice to have.

There is alot of code I'd be happy to kill in Pex that supports this, including a PEP-517 / PEP-518 sdist builder that was needed to get this working.

Is that code different than what was needed for VCS requirements to work properly?

[jsirois]

Is that code different than what was needed for VCS requirements to work properly?

Yes.

[jsirois]

Oof, that was some serious misunderstanding on my part between local requirements vs projects.

You must have been completely mystified by several PRs! You definitely should have spoken up.

[Eric-Arellano]

You must have been completely mystified by several PRs! You definitely should have spoken up.

I understood what the local projects were when reviewing the PRs that actually implemented it. I did not realize that https://github.com/pantsbuild/pex/issues/1696#issuecomment-1086661033 was the reason all that work happened. I apologize for the miscommunication.

There is alot of code I'd be happy to kill in Pex that supports this, including a PEP-517 / PEP-518 sdist builder that was needed to get this working.

I think this is worth considering. It doesn't seem that Pants "needs" local project support, whereas it definitely does want local requirement support. I'm doubtful that we could budget the time in the next 2-3 months for wiring this all up to Pants.

cc @benjyw @stuhood if you have thoughts

[cognifloyd]

By "local project", do you mean "editable installs" (which is only one kind of local project listed at https://pip.pypa.io/en/stable/topics/local-project-installs/)? If you do mean "editable install", then:

When exporting a venv with ./pants export --resolve=... (using the default mutable venv): It would be nice if pants would add an editable install to the exported venv of all the relevant (present in the resolve) python_distributions.

A "local project" might also be interesting if you wanted a sandbox to use local project installs instead of PYTHONPATH manipulation. eg maybe for an adhoc_tool that needs all installed metadata (which goes beyond the code being available on the PYTHONPATH)?

[jsirois]

By "local project", do you mean "editable installs"

Basically - yes. But pip / setuptools need not be in the picture. This simply means locking a local project directory. The only requirement is the local project directory contain a top-level pyproject.toml or setup.py. There are 0 other requirements. The lock is formed by creating an sdist using pyproject.toml / setup.py and then hashing that sdist. If any of the local project files that are relevant (included in the sdist) are altered, consuming the lock will fail and the lock will need to be re-generated to lock the new state of the local project.