ghost commented 5 years ago

Hello, as I am able now to measure the lua used memory of my loaded mods in minetest, (for this may look here) moreblocks is using near 11950 KByte of lua memory (Biggest user on my server). As there are reported OOM crashes, when using LuaJIT is is important to reduce the usage of memory to avoid OOM.

3 Memory use optimisations for Lua mapgens
Wiki - Mapgen memory optimisations and also the using of local var, is speeding up lua also
Lua Performance Tips
Lua object memory sizes as some of this optimization are also speeding up the code itself.

So you know your code better then others, to see what can be changed, as I guess might bring a lot memory saving.

ghost commented 5 years ago

please see Re: [Server] Asia Thailand Minetest 5.0.0 :30001 aktive as detected changes in behaviour with new version.

so what you changed that after all 1.500 KB more mem used now ?

I am working on some automatic lua memory cleaning, better then stop_lj_oom, but still not "able" (lua knowledge) to help you there inside your code

as also twoelk asked: what is happening inside your code to may understand and find solutions

Calinou commented 5 years ago

@Minetest-One Can you test every commit from the latest one (no need to test commits that didn't change any Lua files) to see where the memory usage increase comes from? That would be highly appreciated.

ghost commented 5 years ago

uff, had to search first how to find commit-versions, I just did today my first own mod, and still not sure for what all the links so the one new is d96f110 I just took, the 1d054ec is 3 days older and 0fea5ee 14.07.18 might be the last one to compare, so 3 of them give me some time - as I restart the test-server about 10 times again, again, so see if values keep similar

?? so cant download a commit-version, I don't see it - but should be able to reverse the seen changes at the last one, so then have the older version and the very old should be still here somewhere

ghost commented 5 years ago

here we go: loading:

time (mem to 377),
free_lj_mem (mem to 1004),
moreblocks (mem to see below),
unified_iinventory (end about ~600 higher than moreblocks) so these together just need 1600 during loading, but also I have seen that unified_inventory then index ? collect and rise, but here the value after was lower at old version, and higher at both new ones.

values are: 1st what might mod load reached mem during start, 2nd after unified_inventory showed loaded size, 3rd after that collectgarbage(), and show again

old version 0fea5ee 14.07.18, loaded on 9.3.2019 acts at start 10805 - 8931 - 8931 10825 - 8857 - 8857 10868 - 8970 - 8970 10777 - 8848 - 8848 10763 - 8879 - 8879
newer version d96f110 (so far I did right changes back, but I guess I did) 7078 - 12037 - 12037 7078 - 12037 - 12037 7425 - 12376 - 12376 7529 - 12214 - 12215
latest version 1d054ec (not much idff) 7233 - 12186 - 12186 7363 - 12309 - 12309 7312 - 12256 - 12263 7280 - 12225 - 12225 7240 - 12187 - 12187

like I realised, new version load less at begin, but get huge after, and keep more memory Why that small change cause so big diff, might be interesting to investigate as we have here less change in code but much in memory behaviour

I have no idea yet, and take a break - let this sink into brain ... or bad luck, get lost add: was that really the only change ? in moreblocks code ? I solved that problem secondary - using free_lj_mem to keep memory low. But as a fact, this is one of the most memory using mods I know, end understand code too less yet to understand why. Good luck

fluxionary commented 2 years ago

i don't have anything concrete to add here, but more recent versions of minetest have plugged a number of memory leaks. moreblocks currently creates a huge quantity of new node ids, and the client generally doesn't handle nodeblock/mesh drawtypes as well as full nodes. this is probably an intractable problem to fix within the scope of this mod, though i think we should give links to relevant engine issues before closing this.

fluxionary commented 2 years ago

in my experience - which is broad but far from total across the minetest mod ecosystem - luajit OOM issues are mostly related to worldgen (caverealms, other_worlds, underch, etc.).

191 should help reduce the impact of moreblocks on memory in the future, but overall, this is an architectural issue w/ minetest, and moreso luajit, and so is outside the scope of this mod.

minetest-mods / moreblocks

lua memory usage #140

191 should help reduce the impact of moreblocks on memory in the future, but overall, this is an architectural issue w/ minetest, and moreso luajit, and so is outside the scope of this mod.