bookwyrm-social / bookwyrm

Social reading and reviewing, decentralized with ActivityPub
http://joinbookwyrm.com/
Other
2.17k stars 255 forks source link

possible fix for #3372 - user export timeouts #3378

Closed hughrun closed 1 week ago

hughrun commented 3 weeks ago

Are you finished?

Linters

Tests

What type of Pull Request is this?

Does this PR change settings or dependencies, or break something?

Details of breaking or configuration changes (if any of above checked)

Description

This definitely needs to be tested on a large DB but I believe it may fix the timeouts b.s. gets when running user exports.

Instead of a gigantic single DB query with heaps of joins, we instead just do a series of simple queries and then use union() to pull them into a de-duped queryset.

If I understand the results from explain() correctly, this is a massive reduction in DB work:

Unique (cost=195899.15..198201.71 rows=11808 width=19220)

vs

Unique (cost=150.28..153.44 rows=16 width=19220)

Documentation

mouse-reeve commented 3 weeks ago

I tried this with the query time test system in #3380 and it when from 0.06 to 0.003 which is a huge improvement!