search

kronusaturn / lw2-viewer

An alternative frontend for LessWrong 2.0
https://www.greaterwrong.com/
MIT License
64 stars 6 forks source link

Warning: We are going to limit count queries to 500 for a while #12

Open Discordius opened 5 years ago

Discordius commented 5 years ago

We've been running into a bunch of problems with properly indexing count queries on our database, and GW runs a bunch of really costly count queries on every pageload (or at least every load of the archive) that currently take up to 1s each on our DB, causing a bunch of timeouts for a lot of other queries (and this is the primary reason for the occasional 503 you get from the LW site currently).

Since I don't want to expose API functionality that can take down the main site, the best option for now is to limit count queries to 500, which can safely run unindexed on our DB, while we work on properly indexing the count queries (which will sadly require some significant restructuring of our data and schema because of a bunch of outstanding MongoDB bugs that we have to work around).

This might require some changes on your site, so I figured I would try to warn you as soon as possible. 500 should be high enough that you don't run into any problems on anything but the archive page, where you might have to do some annoying workaround for pagination (either via a caching layer, or just doing queries with skip parameters which should still work as always).

kronusaturn commented 5 years ago

Worked around in 43fe21cb, but this means no more "last page" links.