search

mytakedotorg / mtdo

The code and tooling which runs mytake.org
https://mytake.org
GNU Affero General Public License v3.0
16 stars 3 forks source link

Improve memory consumption #439

Closed nedtwigg closed 3 years ago

nedtwigg commented 3 years ago

We're running at about 15% swap, which is no good. Here's a few things to bring that down.

nedtwigg commented 3 years ago

This PR kinda works, but we'll need to do more. Example memory trace after this PR was merged:

image

Timewise, the problem seems to come when searches are performed. Which is a bit weird, because we try to eagerly warmup the Lucene index:

https://github.com/mytakedotorg/mtdo/blob/50edf1a2e52e274fff1b798e15cb55c22adf73a6/server/src/main/java/controllers/SearchModule.java#L41-L48

Seems like maybe Lucene is doing some caching of its own as it gets used for novel searches. That is bad, because we aggressively cache search results at the HTTP level anyway, so internal caching (if that is what's happneing) is totally redundant.