Introduces BulkVersionUpdater to fix Version read/write N+1s in PackageManager::Base

librariesio / libraries.io

:books: The Open Source Discovery Service

https://libraries.io

GNU Affero General Public License v3.0

1.1k stars 206 forks source link

Introduces BulkVersionUpdater to fix Version read/write N+1s in PackageManager::Base #3268

Closed tiegz closed 3 months ago

tiegz commented 7 months ago

these changes take a cold update of npm/gatsby from 67 sec to 34 sec for me locally (down from 83 min originally, before https://github.com/librariesio/libraries.io/pull/3267)

replace VersionUpdater with BulkVersionUpdater, which does all the version upserts in one query (accompanied by manual callbacks)
remove the uniqueness validation on Version for [:number, :project_id] because we have a unique index on those columns

johnbintz-tidelift commented 3 months ago

Is this still needed?

tiegz commented 3 months ago

I've not had time to figure out how to keep the timestamps updated properly. I'm closing it for now, but if we see performance issues we can always return to it. RIP