Improve publish performance, especially for prefixes with a large number of snapshots

refi64 commented 9 months ago

This contains a variety of improvements for publish performance, specifically speeding up the cleanup operation by:

using a separate, faster package to walk the filesystem
avoiding re-loading the same packages repeatedly (this will happen if a single prefix has a large number of snapshots or repos that share most of their packages)
slightly improving the package list loading performance w/ zero-copy deserialization

Benchmarks were added for all of these, and some unit tests were added specifically to test aspects of cleanup.

We have a relatively large aptly repository with >90 repositories, ~207k packages across all of the repositories, and >3.5k snapshots; a testing version of that repository was used to measure the publishing performance. Prior to these changes, publishing took >9 minutes, with over 8 minutes of that time just in the cleanup phase. With these, the cleanup time goes down to ~13 seconds, for a total publish time of a little under a minute.

Checklist

[x] unit-test added (if change is algorithm)
[ ] functional test added/updated (if change is functional)
[ ] man page updated (if applicable)
[ ] bash completion updated (if applicable)
[ ] documentation updated
[ ] author name in AUTHORS (I already have a commit for this in #1220 and don't want merge conflicts)

codecov[bot] commented 9 months ago

Codecov Report

Attention: Patch coverage is 83.72093% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 66.06%. Comparing base (f1649a6) to head (ae09dfc). Report is 40 commits behind head on master.

:exclamation: Current head ae09dfc differs from pull request most recent head 75b858b. Consider uploading reports for the commit 75b858b to get more accurate results

Files	Patch %	Lines
api/files.go	0.00%	8 Missing :warning:
deb/publish.go	79.31%	4 Missing and 2 partials :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #1222 +/- ## ========================================== - Coverage 74.85% 66.06% -8.80% ========================================== Files 143 143 Lines 16187 16215 +28 ========================================== - Hits 12117 10712 -1405 - Misses 3134 4752 +1618 + Partials 936 751 -185 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.