search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.33k stars 636 forks source link

Performance Tracking: `generate-lockfiles` #21223

Open sureshjoshi opened 3 months ago

sureshjoshi commented 3 months ago

There is a common mention/complaint that calling pants generate-lockfiles :: is slow. Creating this ticket to capture associated discussions, tickets, and even better, sample repos.

From my brief experiments, using pip 24.1 improves cold lockfile generation by about 50% on a small set of requirements, but does not affect incremental checking.

Some ideas that have been floated around:

jsirois commented 3 months ago

Use pex's incremental lockfile updates (available, but not wired into Pants)

This is widely misunderstood. Pex does incremental updates of lock files, but it does this using a full Pip resolve which is exactly as fast as a normal lock create. This bug tracks doing an incremental resolve: https://github.com/pex-tool/pex/issues/2044 These are two orthogonal things.

Use fast-deps if we're not currently doing so

See here, but I tried this and found its still slow: https://github.com/pex-tool/pex/issues/2375#issuecomment-1960010444

sureshjoshi commented 3 months ago

This, bug tracks doing an incremental resolve: https://github.com/pex-tool/pex/issues/2044 These are two orthogonal things.

I was just writing a comment on that ticket asking whether what we have today vs what the ticket refers to were comparable. So "incremental update" vs "incremental resolve" would be a better distinction on the pex side.

See here, but I tried this and found its still slow: https://github.com/pex-tool/pex/issues/2375#issuecomment-1960010444

Ah, that sucks, but yeah - that was just an example I grabbed from one of the many slack/discussions/issues about this.

cburroughs commented 2 months ago

Tracker of several pending pip performance improvements: https://github.com/pypa/pip/issues/12184

cburroughs commented 2 months ago

Sorry, the correct tracker is now: https://github.com/pypa/pip/issues/12921

cburroughs commented 2 weeks ago

Crosslinking some other pip items of possible relevance:

notatallshaw commented 2 weeks ago

It would be very helpful if you can report resolutions that are slow or problamatic on either https://github.com/notatallshaw/Pip-Resolution-Scenarios-and-Benchmarks or https://github.com/pypa/pip.

Probably you would be best switching to uv (which I do also report lots of resolution issues to), but I do aim to keep improving pip's resolver.