Add support for caching pants plugins in CI when using remote cache

[asherf]

In CI scenarios, when using remote cache it is recommended to delete/avoid caching ~/.cache/pants/lmdb_store & ~/.cache/pants/named_cache using the CI system's cache. In CI scenarios the CI system's cache should only be used to cache the pants install itself (~/.cache/pants/setup ) thus preventing the need to bootstrap & download pants for every CI system run. In this kind of scenario the pants plugins (specified under global.plugins) need to be downloaded and "installed" for every CI run. I noticed that this can take up to ~15 seconds (but that obviously depends on the number of plugins used and the packages that need to be downloaded and installed from pypi).

As the pants ecosystem grows we are likely to start having plugins that are external to pants and when using remote cache in CI, installing those plugins becomes somewhat problematic.

@benjyw @stuhood fyi

discussion: https://pantsbuild.slack.com/archives/C046T6T9U/p1639807577351800

[asherf]

In the pants CI downloading/resolving the plugins takes about 16 seconds. which is still a lot considering the size of those packages (toolchain plugin is about ~100kb and the other hdrhistogram is ~60kb).

https://github.com/pantsbuild/pants/runs/5854032908?check_suite_focus=true#step:10:20

@stuhood @Eric-Arellano

[stuhood]

As it stands, the "bootstrap scheduler" which is used to load plugins (and which might hopefully also be used to parse options at some point) intentionally re-uses the ~/.cache/pants/{lmdb_store,named_caches} directories, in order to be able to share cache hits and PIP caches with the actual scheduler used for the rest of the run.

But if caching bootstrap is something that we will want users to put in a local cache with their CI provider, maybe we should move both of those under ~/.cache/pants/setup (or something similar) instead, and then recommend that folks preserve that across runs.

This would be a fairly trivial change to make: it's mostly about figuring out how to document it and what to recommend here: https://www.pantsbuild.org/docs/using-pants-in-ci ... since it actually ends up adding complexity (at least superficially).

[benjyw]

Possibly having a bootstrap location just the pex_root named cache is enough in practice?

[asherf]

As it stands, the "bootstrap scheduler" which is used to load plugins (and which might hopefully also be used to parse options at some point) intentionally re-uses the ~/.cache/pants/{lmdb_store,named_caches} directories, in order to be able to share cache hits and PIP caches with the actual scheduler used for the rest of the run.

But if caching bootstrap is something that we will want users to put in a local cache with their CI provider, maybe we should move both of those under ~/.cache/pants/setup (or something similar) instead, and then recommend that folks preserve that across runs.

This would be a fairly trivial change to make: it's mostly about figuring out how to document it and what to recommend here: https://www.pantsbuild.org/docs/using-pants-in-ci ... since it actually ends up adding complexity (at least superficially).

I think this is somewhat niche, since there are not a lot of 3rd party plugins out there, I assume that if people use plugins they are mostly 1st party (in their own repo), so the impact of this change is somewhat limited and should be prioritized accordingly. (I know I opened the ticket, but wanted to give more context from my POV).

[kaos]

FWIW our use of plugins at work is installed as 3rd party in order to be re-usable across multiple repos.