librariesio / libraries.io

:books: The Open Source Discovery Service
https://libraries.io
GNU Affero General Public License v3.0
1.1k stars 204 forks source link

feat: [sc-31318] NuGet project verification worker: fix queueing adjust and rate of work #3224

Closed macowie closed 9 months ago

macowie commented 9 months ago

Story details: https://app.shortcut.com/tidelift/story/31318

Fix spacing out the batches, and slow the rate some more to be safe & considerate.

macowie commented 9 months ago

Would it be worth putting in something to let the user pick up where things started breaking partway through a run instead of going through the entire dataset each time?

Wasn't sure how best to go about this, but this seems to work and could be a good pattern going forward.