Slow Queries for Pagination with Using Limit / Offset in Large Sets

[michaelhiiva]

Summary of the Bug

@duckpunch https://github.com/usgo/agagd/pull/180#discussion_r615553918

What's the rough size of the largest thing we'll pass in? LIMIT/OFFSET can be pretty slow, right?

Specifically the issue with performing large queries on large sets. https://docs.djangoproject.com/en/3.2/ref/paginator/#django.core.paginator.Paginator.object_list

If you’re using a QuerySet with a very large number of items, requesting high page numbers might be slow on some databases, because the resulting LIMIT/OFFSET query needs to count the number of OFFSET records which takes longer as the page number gets higher.

Notes

• https://thetldr.tech/how-to-implement-pagination-in-django-using-limit-and-offset/
• https://docs.djangoproject.com/en/3.1/topics/db/queries/#limiting-querysets.
• The issue applies to beta right now.