search

taoensso / nippy

The fastest serialization library for Clojure
https://www.taoensso.com/nippy
Eclipse Public License 1.0
1.04k stars 60 forks source link

Is there a written out spec for the binary format? #147

Closed bowbahdoe closed 11 months ago

bowbahdoe commented 2 years ago

Forgive me if i'm just not finding it, but is there a spec for the binary format similar to https://download.java.net/java/early_access/panama/docs/specs/serialization/index.html ?

I'm looking to implement a binary EDN sort of thing in Rust and I don't see any reason to re-invent a data format

ptaoussanis commented 2 years ago

@bowbahdoe Hi Ethan,

No, there's unfortunately no formal written spec at the moment. But the format is stable, reasonably simple, and shouldn't be too difficult to understand from reading the codebase.

The details depend very much on what your objectives are. Would your intention be to actually have binary data compatibility between Rust and the JVM? Or you're just looking for inspiration?

Binary data compatibility would likely be non-trivial to tough [1]. Nippy was never designed to be a format portable between platforms. Its design tradeoffs emphasise performance and simplicity of [JVM] implementation over portability. If the goal is portability, a different approach may be sensible.

If you let me know in more detail what you're hoping to achieve, I can try advise better.

Cheers!

[1] The meat of Nippy's data format is actually provided by java.io.DataOutput and java.io.DataInput. Nippy's mostly a thin wrapper around these - mostly to add Clojure type information. Some limited use is also made of java.io.Serializable. So the difficulty of de/serializing Nippy data on another platform will depend mostly on whether your target platform already has [/library] support for the relevant java.io.DataOutput and/or java.io.DataInput formats.

bowbahdoe commented 2 years ago

I feel almost embarrassed because of the relatively low chance I produce anything useful, but i'm aiming to make an open source Datomic clone in rust.

The only open source datomic clone I know of is the unmaintained EVA and it uses fressian for storing its datoms. XTDB uses nippy. I just don't want to invent something brand new.

So lets say the goal is "to support a subset that nippy could read, but not support everything nippy could write"

ptaoussanis commented 1 year ago

@bowbahdoe Hi Ethan, just following up on this. [com.taoensso/nippy "3.3.0-alpha2"] now contains a new var that includes some basic info re: Nippy's types.

Not sure how helpful this'd be in your case, since I expect that the bulk of the work for getting something a Rust implementation working would be in matching the java.io.DataOutput/java.io.DataInput formats.

In any case, the type spec is now more readable - so should hopefully give you a better idea of what's involved.

But please do let me know if you think of anything else that I/Nippy might be able to provide to make your objective easier.

Cheers!

bowbahdoe commented 1 year ago

@ptaoussanis Thanks a lot man! This has circled in the infinite backlog of my stuff so it will be a bit before i give it another go, but this is useful

ptaoussanis commented 11 months ago

Closing to focus discussion at https://github.com/taoensso/nippy/issues/151