Improve performance - Githubissues

maximusunc commented 1 year ago

This PR is aimed at improving the performance and speed of Strider.

Testing a creative mode query with 83 sub-queries
- No Cache: In the previous version, it was approximately 97 minutes to complete all queries, with the new it takes about 20 mins
- With Cache: previous version I cut it off at 45 mins but I expect it would have taken at least another 15. With the new, it takes about 8 minutes.
We are now compressing all the entries in redis. This will cut down our storage footprint by ~5x. The plan for deployment is to also bump the redis container up from 10GB to 100GB? With that, we could probably also bump up the TTL on each key from 1 week to something a little longer? @cbizon @kennethmorton do you guys have thoughts on this?

kennethmorton commented 1 year ago

Make sure tests are good and then let's merge it. The performance upgrade is huge!

cbizon commented 1 year ago

So if the redis runs out of memory does it chuck the oldest things first? Assuming it does, then yeah, I think you could up TTL to twice or 4 times its current amount.

maximusunc commented 1 year ago

I have redis key eviction set up to use volatile-lfu: https://redis.io/docs/reference/eviction/

uhbrar commented 1 year ago

It's also very concerning that the tests are taking 5+ minutes to run, and I suspect they won't actually finish. Something is going on here.

ranking-agent / strider

Improve performance #405