Closed jsirois closed 11 months ago
This issue is framed pretty prescriptively; so extra care should be taken to question assumptions. The main points are to simplify and robustify distribution and consumption of Pants while taking financial and operational burden away from OSS entities and push that towards OSS benefactors like Microsoft.
The idea proposed here is releasing Pants as a PEX on GitHub releases. Later, also releasing the Pants client binary as well. These releases could be consumed in 2 obvious ways:
The second approach is appealing for separation of concerns. The scie-pants binary would only need to know a cut-over version for PyPI wheel -> GitHub Releases scie. Pants could then evolve its install and execution logic separately. The scie-pants binary then just handles invoking the right Pants version and ~nothing more. It stays simple and very robust while allowing Pants to add complexity and nuance on a per-release basis without worry of breaks cross-project. Towards this end, Pants need not even release a scie, it could just release a lift manifest + PEX (+ eventually native client). The lift manifest would point to the PEX (and eventually native client), locking in URL / size / checksum, as well as install and execution logic for that release and the scie-pants
binary could just download and verify the checksum of the lift manifest and use it to locally assemble a Pants scie to delegate to in the hot path. Here https://github.com/a-scie/jump/issues/10 would be useful and allow scie-pants
to not even build a scie at all, but just "execute" the Pants lift manifest.
Whether shipping a PEX to Github or a scie to github, this would also involve pants_requirements growing the ability to load itself from Pants' own runtime environment... or perhaps scie-pants
could execute a pre-extraction step which continued to use find-links
, but from within a nce
directory.
@thejcannon has done a ton of work in this area. Maybe worth summarizing what is remaining to do here.
Not sure about work, but when thinking about what we gain I think the big victory would be diminished now to:
There's likely other secondary benefits but I think that's the one big one (and is less beneficial since the actual support isn't really that challenging)
I'll just close this as obsolete. I don't think it was used as guidance for the migration and the migration is nearly complete.
Currently, and for most of its life, Pants has been distributed as one or more Python projects distributed on PyPI. Pants v2 has settled on the
pantsbuild.pants
project to deliver thepants
console script and thepantsbuild.pants.testutils
test infrastructure project for writing plugins whose code lives outside the https://github/com/pantsbuild/pants repo. The./pants
script has been checked into Pants-using repos and used to installpantsbuild.pants
in a venv and forward to itspants
console script as the primary means of installing and running the right version of Pants in the right way in a repo. Recently, migration from the./pants
script to thescie-pants
binary has begun.The
scie-pants
binary works with thepantsbuild.pants
PyPI project to maintain the status quo for installing and executing current and older Pants version but it also provides an avenue to alter:It's desirable to alter 1 - our primary PyPI distribution - to take load off PyPI's storage and transfer costs. We're currently in the top 50 users of storage space (31st using 36.5GB as of today, but see: https://pypi.org/stats/). Its desirable to alter 2 - our self-paid-for S3 bucket for similar reasons but for the Pantsbuild non-profit organization itself.
If we alter 3 - Pants assembly - to a single or few file format, we can cut down on flaky transfers (one or few connections tends to be better in this regard) and gain easier insight into what makes up a Pants release. The natural alternative here is distribution as a PEX. Pants already does release a PEX via (GitHub Releases](https://github.com/pantsbuild/pants/releases), but it seems to be a little used facility. This is probably partially due to the current PEX being very large (it's multi-platform), partially to the https://pantsbuild.org docs not recommending it front and center and partially due to it being non-optimal - it adds latency over a pure venv.
The scie-pants launcher could be altered to handle Pants released as a PEX on GitHub releases since it implements complex install logic efficiently and isolated from the hot path of a Pants run. One way of doing this would be to release a Pants.pex per supported platform (today: {Linux, Mac} x {x86_64, aarch64} along with a sha256 checksum of each of these PEXes. This would obviate the need for either PyPI (1) releases or pantsbuild.binaries.org (2) releases and - currently - cost the Pantsbuild organization nothing since GitHub releases are currently free without limits for OSS projects. To be clear, there is no magic here, we'd be getting away with murder and costing the world just as much in storage and transfer costs; we'd just be letting Microsoft foot the bill until we're forced to reconsider.
If we transition to only releasing Pants as a PEX that is installed by
scie-pants
we can eliminate the pantsbuild.pants wheels and just directly add the Pants code to the PEX via-D src/python
. This eliminates some BUILD jank partially tracked in #7344. It also allows us to eventually ship the Pants native client as a separate binary with sha256 checksums via GitHub Releases and use it as the hot-path executable scie-pants invokes.