Make MultiServer use less memory

black-sliver commented 2 years ago

TLDR

Currently we are looking at roughly 20MB per instance. With each world added this will rise. With rising popularity per world this may become exponential.

Analysis

MultiServer.py loads all the worlds and keeps references. This happens in

from worlds.AutoWorld import AutoWorldRegister

proxy_worlds = {name: world(None, 0) for name, world in AutoWorldRegister.world_types.items()}

and can't be undone easily because each world will keep a reference to AutoWorld, which keeps a reference to all worlds.

Saving potential

MultiServer.py only needs a world reference for very few things currently, so it could instead work on the same DataPackage clients use with some extras, not loading any world. Or it could load only the played worlds, which would still save the majority of the memory. Currently the client protocol requires each MultiServer to hold data for all worlds, but that will be changed in #813 via #757

Proposed solution A

Generate and cache DataPackage, include extra info, like Groups, which are not part of DataPackage but accessibel by the server.
Load DataPackage + extra info instead of world references
:white_check_mark: Change all lookups to use that data
Once old protocol compatibility is dropped, delete unused worlds from data (dicts).

edit: Points 1 and 2 are done for WebHost, point 3 should be complete in #877 Point 4 is new

alwaysintreble commented 2 years ago

I think since the plan is to add DataPackage to the multidata anyways adding additional server information to it might be better? I might be able to refactor some of the tracker info to be passed in this way also instead of needing to pull the info from the worlds at runtime.

black-sliver commented 2 years ago

Berserker didn't want to have the complete datapackage in multidata, only the extra keys for slot-specific locations/items. It may be worth considering to have the datapackage in there so that datapackages don't have to be backwards-compatible anymore, it would make it impossible to fix bugs in datapackage for seeds that are already rolled.

Afaik a rework was already started. Maybe @Berserker66 can post the current plan here.

Berserker66 commented 2 years ago

The rework is independent of those ideas for now. I have some additions of defaultdicts to make it easier to have them fall back to empty data if a game is unknown, but that's about it. And yes, the plan is not to include the entire datapackage, but try to be smart about what is dynamic or newer than target.

ArchipelagoMW / Archipelago