harvard-lil / perma

Indelible links
417 stars 71 forks source link

Tweak strategy for finding links to upload to IA #3451

Closed rebeccacremona closed 10 months ago

rebeccacremona commented 10 months ago

While investigating various strategies for monitoring slow or expensive queries, it came to @mdellabitta's attention that the job to enqueue internet archive upload tasks, which runs every five minutes, was using up more DB juice than we'd like.

This PR takes another swing at making that query less expensive.

First, it uses different logic, depending on the case:

Second, it reorders the queries to use a new index, an index that starts with the "date" portion of Link.creation_timestamp.

Tested locally with a production-like database, performance did improve: I can see the new index being used, and can see times decrease..... but, on my laptop, these queries are never particularly slow.

New: image

Orig: image

Let's see if this helps enough... If it doesn't, we might price out getting some more RAM.

With thanks to @mdellabitta and @bensteinberg for suggestions and moral support when debugging the index creation.

codecov[bot] commented 10 months ago

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (d73d2f0) 70.41% compared to head (2bf0ceb) 70.42%. Report is 9 commits behind head on develop.

Files Patch % Lines
perma_web/perma/models.py 22.22% 7 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #3451 +/- ## =========================================== + Coverage 70.41% 70.42% +0.01% =========================================== Files 48 48 Lines 6568 6567 -1 =========================================== Hits 4625 4625 + Misses 1943 1942 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.