osuAkatsuki / bancho.py

An osu! server for the generic public, optimized for maintainability in modern python
https://akatsuki.gg
MIT License
217 stars 131 forks source link

misc: reduce memory usage of performance recalculation script #412

Open TrueRou opened 1 year ago

TrueRou commented 1 year ago

Cache every map during recalculation may cause memory lack on low-end server if there are too many maps (may eat many gigs of ram). It's better to make cache optional which can easily enable or disable by providing args (--use-cache or other something)

tsunyoku commented 1 year ago

maybe too high effort solution but we could use a cache which keeps the most frequently accessed maps and evicts the others to keep the memory down

minisbett commented 1 year ago

maybe instead of recalculating pp on a score id going upwards basis calculate it per map? so it goes through all maps in the maps table and recalcs all scores for that particular map?

TrueRou commented 1 year ago

maybe instead of recalculating pp on a score id going upwards basis calculate it per map? so it goes through all maps in the maps table and recalcs all scores for that particular map?

I agree with that, it just like the old way we calculate by command.

cmyui commented 1 year ago

is this query the problem?

https://github.com/osuAkatsuki/bancho.py/blob/701c4627cf66c4dd5d5f364ef35e7cd7cb53423a/tools/recalc.py#L183-L190

looks like it's only chunked for processing once in memory which is not particularly useful, it should be pulled from sql in chunks to keep memory usage down

minisbett commented 1 year ago

The problem is that everytime a map gets recalced the whole map needs to be read by rosu-pp and therefore loaded into ram I think

TrueRou commented 1 year ago

is this query the problem?

https://github.com/osuAkatsuki/bancho.py/blob/701c4627cf66c4dd5d5f364ef35e7cd7cb53423a/tools/recalc.py#L183-L190

looks like it's only chunked for processing once in memory which is not particularly useful, it should be pulled from sql in chunks to keep memory usage down

This won't take too much because millions of scores will only cost about 100 mb. It's worthwhile to cost memory in enchange of some io performance