Versioning

[slav]

Protobuf supports versioning by assigning number to each field. As long as type and number matches, serialized data can be deserialized correctly. I can rename the actual field, move it around, add new fields (which would have default value if serialized data didn't have the field value).

Is it possible to achieve the same with FsPickler? Or as soon as I change schema things would break?

Also, have you tried to test serialized size comparison of FsPickler to other serializers?

[eiriktsarpalis]

I'm not familiar with this. Sounds kind of similar to the DataContract pattern (which FsPickler already supports), where you can (optionally) assign an order identifier to each serialized property by way of attributes. I think this could be implemented easily in FsPickler.

[eiriktsarpalis]

I hadn't actually thought of making size comparisons before, here are some preliminary results of a test I just knocked together. Primitives are considerably larger in FsPickler due to a type signature prefix in the header. In bigger and more complex objects, sizes are comparable to ProtoBuf-Net.

integer (FsPickler.Binary): 28 bytes
integer (FsPickler.Json): 53 bytes
integer (BinaryFormatter): 54 bytes
integer (NetDataContractSerializer): 171 bytes
integer (Json.Net): 1 bytes
integer (ProtoBuf-Net): 2 bytes
integer (ServiceStack.TypeSerializer): 1 bytes
pair (FsPickler.Binary): 75 bytes
pair (FsPickler.Json): 116 bytes
pair (BinaryFormatter): 269 bytes
pair (NetDataContractSerializer): 500 bytes
pair (Json.Net): 34 bytes
pair (ProtoBuf-Net): 15 bytes
pair (ServiceStack.TypeSerializer): 28 bytes
int list (1000 elements) (FsPickler.Binary): 3.98 KiB
int list (1000 elements) (FsPickler.Json): 3.90 KiB
int list (1000 elements) (BinaryFormatter): 17.98 KiB
int list (1000 elements) (NetDataContractSerializer): 39.35 KiB
int list (1000 elements) (Json.Net): 3.80 KiB
int list (1000 elements) (ProtoBuf-Net): failed
int list (1000 elements) (ServiceStack.TypeSerializer): failed
list of pairs (1000 elements) (FsPickler.Binary): 12.69 KiB
list of pairs (1000 elements) (FsPickler.Json): 27.25 KiB
list of pairs (1000 elements) (BinaryFormatter): 41.11 KiB
list of pairs (1000 elements) (NetDataContractSerializer): 166.48 KiB
list of pairs (1000 elements) (Json.Net): 27.13 KiB
list of pairs (1000 elements) (ProtoBuf-Net): failed
list of pairs (1000 elements) (ServiceStack.TypeSerializer): failed
3D float array (100x100x100) (FsPickler.Binary): 3.81 MiB
3D float array (100x100x100) (FsPickler.Json): 7.51 MiB
3D float array (100x100x100) (BinaryFormatter): 3.81 MiB
3D float array (100x100x100) (NetDataContractSerializer): failed
3D float array (100x100x100) (Json.Net): 7.53 MiB
3D float array (100x100x100) (ProtoBuf-Net): failed
3D float array (100x100x100) (ServiceStack.TypeSerializer): failed
exception with stacktrace (FsPickler.Binary): 2.30 KiB
exception with stacktrace (FsPickler.Json): 3.08 KiB
exception with stacktrace (BinaryFormatter): 2.00 KiB
exception with stacktrace (NetDataContractSerializer): 2.78 KiB
exception with stacktrace (Json.Net): 2.02 KiB
exception with stacktrace (ProtoBuf-Net): failed
exception with stacktrace (ServiceStack.TypeSerializer): 5 bytes
F# quotation (FsPickler.Binary): 15.14 KiB
F# quotation (FsPickler.Json): 42.42 KiB
F# quotation (BinaryFormatter): 22.31 KiB
F# quotation (NetDataContractSerializer): 119.03 KiB
F# quotation (Json.Net): failed
F# quotation (ProtoBuf-Net): failed
F# quotation (ServiceStack.TypeSerializer): 575 bytes
Single Record (FsPickler.Binary): 187 bytes
Single Record (FsPickler.Json): 316 bytes
Single Record (BinaryFormatter): 342 bytes
Single Record (NetDataContractSerializer): 651 bytes
Single Record (Json.Net): 238 bytes
Single Record (ProtoBuf-Net): 130 bytes
Single Record (ServiceStack.TypeSerializer): 218 bytes
Dictionary of Records (1000 entries) (FsPickler.Binary): 146.17 KiB
Dictionary of Records (1000 entries) (FsPickler.Json): 265.56 KiB
Dictionary of Records (1000 entries) (BinaryFormatter): 176.93 KiB
Dictionary of Records (1000 entries) (NetDataContractSerializer): 686.67 KiB
Dictionary of Records (1000 entries) (Json.Net): 238.84 KiB
Dictionary of Records (1000 entries) (ProtoBuf-Net): 135.62 KiB
Dictionary of Records (1000 entries) (ServiceStack.TypeSerializer): 217.67 KiB
Binary tree (balanced, depth = 10) (FsPickler.Binary): 12.92 MiB
Binary tree (balanced, depth = 10) (FsPickler.Json): 23.95 MiB
Binary tree (balanced, depth = 10) (BinaryFormatter): 30.16 MiB
Binary tree (balanced, depth = 10) (NetDataContractSerializer): 39.78 MiB
Binary tree (balanced, depth = 10) (Json.Net): 23.37 MiB
Binary tree (balanced, depth = 10) (ProtoBuf-Net): 11.49 MiB
Binary tree (balanced, depth = 10) (ServiceStack.TypeSerializer): 21.18 MiB

[slav]

DataContract and DataMember with Order set to a number is exactly what I'm talking about. Although, from what I've tested it's impossible to easily configure discriminated unions cases and each case members with attributes to specifying order.

You mentioned that FsPickler already supports DataContract pattern and order identifier? How? Or is it still needs to be implemented.

FsPickler looks very promising, but versioning needs to be supported in some form, otherwise sooner or later things will begin to break.

Thank you for posting size comparison.

[eiriktsarpalis]

It is already implemented, you can have a look at it here:

https://github.com/nessos/FsPickler/blob/master/src/FsPickler/PicklerGeneration/DataContractPickler.fs#L48

I should add that FsPickler does not tolerate missing fields, thus adding new properties would almost certainly break the serialization format.

[slav]

Is there any way to work around it? For example provide a default value? Very often I would store serialized data as immutable piece of data, but the schema does evolve over time, usually with new fields being added.

Also, it would be nice if you could add examples of using DataContract with different F# types (I'm especially interested to see it with discriminating unions).

[slav]

Maybe I should explain what I'm trying to do. I have a series of events, for example

 type ItemEvent =
     | ItemCreated of Id: long * Sku: string
     | ItemAdded of Id: long * Quantity: int * Location: string
     | ItemRemoved of Id: long * Quantity: int * Location: string

I save all events in eventstore. I'd like to be able to serialize any of the union cases and then deserialize. I need to be able to add fields in the future, for example Code: string to ItemCreated.

Would FsPickler be a good fit in this case to store serialized and later deserialize when the schema has changed. I could provide custom code for the changes, for example default value or something else, if it's possible.

[eiriktsarpalis]

The short answer is that the library is not designed to accommodate evolving schemata. This means that you have to compensate by writing your own serialization logic, either by implementing the ISerializable interface or using a custom pickler definition. See the tutorial for more information on the latter.

This is a deliberate design decision. The library was written with distributed computation in mind, so it tends to focus on ephemeral message passing between homogeneous processes rather than persistable, evolving information. It also relates to the fact that the library is based on pickler combinators, which generally speaking generate rigid serialization formats.

[caindy]

@eiriktsarpalis thank you for your thoughtful responses. I had nearly the exact same questions about the applicability of FsPickler.

@slav did you find or develop a better approach? The workaround I've adopted at this point is to use the BCL binary serializer and only append cases to the union. It means having something like ItemAddedV2 added to the end of the union, as opposed to writing a custom serializer, but the stratagem is robust in the face of new cases, afaict. There are plenty other changes that aren't supported, but event stores are about immutability after all.

Apologies to the maintainers for initiating a discussion that is off-topic. @slav is you aren't disinclined, would you reach out on Twitter? I'm @caindy

[slav]

@caindy I ended up using protobuf with custom logic for serializers. Don't really use twitter much, sorry.