search

Cysharp / MemoryPack

Zero encoding extreme performance binary serializer for C# and Unity.
MIT License
3.29k stars 193 forks source link

Circular References - depth limit? #149

Closed glandeurlessard closed 1 year ago

glandeurlessard commented 1 year ago

Hi,

I have a very complex data structure with many objects holding references to other objects (and reciprocally). All serializable objects have the [MemoryPackable(GenerateType.CircularReference, SerializeLayout.Sequential)] tag.

Memorypack crashes most of the time trying to serialize it, and I've saved traces of the process to try to figure out why. Maybe I don't understand how things work but it looks like the same objects are being serialized 100s, sometimes 1000s of times?

image

Any pointers appreciated!

And thanks for the great software.

Jonathan

neuecc commented 1 year ago

Okay, it looks like a circular reference. Is there some special Object Equality setting? Can you give me a reproduction code?

glandeurlessard commented 1 year ago

So I ended up succeeding by "flattening" most of the objects, replacing references by IDs before serializing.

I think the circular references simply went too deep and probably exploded memory. The serializer would enter one object and crashed before reaching the end of it, crawling recursively all references.

I cannot reproduce the problem in a simple scenario, but I could show you the project if you want to look into it.

This is what my debug file looks like when it spirals out and crashes:

image

neuecc commented 1 year ago

please projects.

glandeurlessard commented 1 year ago

In case you want to follow up, I've invited you to the repo.

To replicate:

  1. Checkout the MemoryPack branch
  2. Open the Unity Project
  3. run the "game" scene
  4. When ready, click "explore world freely"
  5. Press the pause button at the bottom of the screen
  6. Press "S" key to start saving

The saving is on a thread so the game might not crash, but the process will likely not finish.

You will find the debug output file in the Unity persistent data path: (on windows) /User/Appdata/LocalLow/LabLabLab/ChronDebug.xml

The xml structure allows you to look at the nesting of the objects.

The saving code is in World.cs : SerializeThis() I have commented the DeflateEverything() function which I wrote to replace references by IDs prior to serializing I have also commented many [MemoryPackIgnore] tags which make sure the process doesn't crash.

Happy to answer any questions.