Exception When Trying to Serialize a Bigger Graph Object

[ab-tools]

As written in the protobuf-net project already, I can reproduce an exception when trying to serialize a bigger (about 50 MB) graph object with AqlaSerializer.

To reproduce it I need to include several classes from my project which I don't want to publish here. Therefore I allowed myself to send you an e-mail to the address I found on your "Aqla Solutions" website.

Please just start this example project, click on "Load Data With Protobuf" and after it's done, click on "Save Data With AqlaSerializer" => Exception.

I really would like to get rid of Protobuf as this project seems to be dead. So I'm very much looking forward to your reply!

Thanks in advance Andreas

[AqlaSolutions]

Hi, Andreas! I checked your issue.

1. It looks like the original protobuf-net can't save your object (the same recursion exception).
2. I fixed the problem with AsReferenceDefault not being used correctly.
3. The serialization goes in a recursive way. Your objects have a similar to linked list structure. That means the whole list is processed in a recursive way which leads to StackOverflowException. I don't see how it can be fixed right now as the architecture assumes depth-first graph traversal.

[ab-tools]

Hello,

first thanks a lot for your very quick reply!

1. Yes, that's correct - the reason I'm searching for another serialization project. ;-)
2. A very good start! :-)
3. I understand that protobuf-net has problems with it as it obviously does not handle object references correctly. But as you fixed that I don't understand why the StackOverflowException when serializing graphs cannot be fixed: you can just "stop processing" when you reach the same object (= reference equal) again.

I just build this small example project now to reproduce it more easily without my database: http://www.ab-tools.com/temp/AqlaSerializerTest.zip

It works fine if you reduce the node count to e. g. only 1000, but with 10000 you get the StackOverflowException. So it obviously works fine also with recursion as long as not too many objects are involved.

Best regards and again thanks for the quick response Andreas

[AqlaSolutions]

@ab-tools

3. "you can just "stop processing" when you reach the same object" this is already done but it's not your case. Your serialization path looks like Airport => Waypoints => Airports => ... Airport => Waypoints => Airports ... Because objects graph is traversed not in wide but in depth. Of course similar references are not traversed second time but it doesn't help when there are objects referencing other different objects referencing other different objects ... The whole "list" is traversed recursively.

There is a workaround but you need to change your structure: remove Waypoint.Airports from serialization but add Dictionary of Waypoint, Airport[] somewhere in the root instead so there will be no more "Waypoints => Airports" part.

[ab-tools]

Yes, I know - that's how I do it with protobuf-net right now (and then iterating through all Airways at program start and adding the Waypoint => Airways references manually again), but I really would like to get rid of all these work-arounds.

I guess I would need less work-arounds than with protobuf-net with your serializer already (due to the reference issue to be fixed), but it would be great to have a serialization solution that just can handle a graph structure correctly without any work-arounds. ;-)

You marked it for V2: when you expect to have a first version of V2 to be usable?

Agains thanks for the very quick replies! Andreas

[AqlaSolutions]

V2 is a "big bright future". I plan to rewrite core things but it's really big task and is not going to happen soon. I wouldn't wait for that :)

[ab-tools]

OK, see - that's bad.

But anyway, AqlaSerializer seems to be the best serializer right now as protobuf-net obviously is dead.

So I'll probably switch to your solution for this project. :-)

Thanks again for this great support Andreas

[AqlaSolutions]

@ab-tools I have another workaround for you. Just create a separate thread with big stack size.

new Thread(s => Serialize(), 1024 * 1024 * 50).Start();

Download the newest release otherwise you'll get recursion exception.

[ab-tools]

That's a good idea, thanks for this suggestion!

I just gave it a try: you need to do the same for deserialization, too, of course, but then it works fine. The application does require full trust then though (for changing the stack size of the new thread), but as my application should not be run over network or similar, I think that's OK (although I need to test a bit more with it as I never needed to change the stack size of a thread till now ;-) ).

I will add a suggestion/feature request in another issue now, but again thanks for your great support regarding this issue here Andreas