search

raiguard / RecipeBook

Recipe Book for Factorio.
MIT License
20 stars 9 forks source link

Script.dat bloat/High memory usage #56

Closed oorzkws closed 3 years ago

oorzkws commented 3 years ago

Describe the Bug

When using mods that result in a significant amount of recipes, Recipe Book will use a large amount of memory and increase the save time to a bothersome amount. To illustrate, script.dat from my personal save is 60mb without RB but 230 with.

To Reproduce

Steps to reproduce the behavior:

  1. Install mods with a high amount of recipes, such as full pysuite plus omnicompression
  2. Install recipebook
  3. Enter game and observe save times
  4. Remove recipebook
  5. Enter game and observe save times

Save file & Username

RecipeBookBugReport.zip Username is _-_-_

Log file

factorio-current.log

raiguard commented 3 years ago

170mb seems like more than it should be, I'll take a look. Recipe Book stores a ton of data but it shouldn't be THAT bad.

raiguard commented 3 years ago

I loaded up Py and Omni, and did some checks. That modset has 29 THOUSAND RECIPES. That's why there is so much global data. Recipe Book keeps data on every recipe, and if you have a lot of recipes, there's going to be lots of data.

There's nothing I can do about this. Sorry.

raiguard commented 3 years ago

image

This is just a lot of data for RB to handle. It's just going to take a while to save and load it all.

raiguard commented 3 years ago

Well actually, I might be able to do something about this after all. I'm experimenting with only keeping the bare minimum amount of data in global, and generating the RB data for each client individually when they load the save. It's looking promising.

raiguard commented 3 years ago

Alright, drumroll please... I made the changes and they worked flawlessly! After solving some dependency loops, everything just worked like a charm.

Before: 102 MB After: 9.3 MB

There's still going to be a non-trivial amount of data, especially with a modset this large. RB still has to store translations for every single thing it has data for in the script data. But the translations take up a small portion of the overall space, so it's still a >10x improvement.

Thanks for bringing this to my attention!

raiguard commented 3 years ago

v3.0 includes massive data storage optimizations, so this issue can now be closed.

oorzkws commented 3 years ago

Wow, thank you giving it a try. That's made a very large difference in my save times! ❤️