rikimaru0345 / Ceras

Universal binary serializer for a wide variety of scenarios https://discord.gg/FGaCX4c
MIT License
482 stars 53 forks source link

Serialization format #21

Open yallie opened 5 years ago

yallie commented 5 years ago

Hello Riki, your serializer looks very promising!

I'm wondering whether its physical format stable enough to be standardized? Would be great to have a wire-compatible javascript version of the .NET serializer. Existing formats like Json, Bson or MessagePack are quite limited (don't support circular references, etc).

Regards, Alex

rikimaru0345 commented 5 years ago

That is a great idea. I was thinking about that for quite some time actually.

Like documenting how exactly the format works in detail so other people can make libraries for other languages. It would be great.

Actually the documenting part would certainly not be that hard to do actually.

But as you already pointed out, there's one important thing to consider, "stability". As in, if there are any large scale changes in the future that'd be really bad and screw over anyone who wants to invest some time to create their own serializer in another language.

Now, I think the format is almost stable right now. There are many things that have changed in the past, but nowadays I think I'm really closing in on something concrete. Like there probably won't be many more large binary format changes in the future.

However I can't exclude that right now. There are still a few tiny things that could cause a need for format changes.

For example what I'm thinking about right now is improving the way schema embedding works for version tolerance. (a way where users can manage the embedded schemata themselves). Maybe it would be good to add a sort of flag into the binary for that (telling ceras to deserialize the data it needs an external schemata-description from the user?) Or maybe not embedding that at all? I'm currently thinking about the implications of either solution (or if there's even other solutions). And that's just one example of potential future changes. But it's not a big one, and there aren't all that many other things left that could require a format change.

All in all I can say this:

That being said, I don't want to lock down the format right now because I want to still have the freedom to really improve things and not think about other people relying on the format not changing.

Ah, and there's also always the possibility of evolving the format itself with version numbers. Just like .net standard has its 1.0, 2.0, ... versions.

rikimaru0345 commented 5 years ago

Also thanks for the nice comment. Turns out publishing a project you worked very hard on is much more stressful than I imagined haha.

yallie commented 5 years ago

I don't foresee any large changes in the future. However I can't exclude the possibility of a few minor changes though.

Ok, got it! Thanks for your detailed reply.

I like the approach taken by ZeroMQ community. They start any new projects by drafting easy-to-read RFC documents describing the protocols. The spec and the reference library evolve in parallel and reach the stability at some point. Once the RFC is marked as stable, the library can be safely ported to another platform.

The point is that an RFC is much easier to follow than the source code of any real implementation. It surely sounds like an overhead for the developer, but it pays off when it comes to porting.

Regards, Alex