kyle1373 / Archiverse

The biggest Miiverse archive on the internet
https://www.archiverse.app
MIT License
14 stars 0 forks source link

Fix gateway timeout errors in searching for posts #30

Closed kyle1373 closed 3 months ago

kyle1373 commented 3 months ago

When calling /api/posts?search=some_query_here, there are many times where users get a timeout error. Figure out why this is the issue and if there is a way to fix it.

kyle1373 commented 3 months ago

So I went ahead and combined the two indexes for Text and Title into one GIN index. It seemed to have sped up the queries, although there are still cases where gateway timeouts occur (usually when there is a cache miss). I think I may need to swap to elastic search to get this working properly. For now, I'll put this issue into limbo, as it's now considered low priority. If this gets bad enough, I'll put more attention to it (and may actually remove post search if CPU utilization gets too high)

kyle1373 commented 3 months ago

I have taken out the search posts endpoint due to the large computational costs on the database for searching for posts. This feature will no longer be available unless we switch to a better plaintext searching framework. Until then, this issue will remain stale.

kyle1373 commented 3 months ago

The best course of action here is to use OpenSearch and creating a digital ocean managed db. Here's what I'll do:

I'll use the .sql file I have on my local machine to construct the db locally. Then, I'll port the data over to opensearch database. Costs should be $30 / mo to keep opensearch running. Might be too expensive to maintain but if the site becomes sustainable with ads then I'll do this.

kyle1373 commented 3 months ago

Current blocker: run ads on site