search

scala / pickling

Fast, customizable, boilerplate-free pickling support for Scala
lampwww.epfl.ch/~hmiller/pickling
BSD 3-Clause "New" or "Revised" License
831 stars 79 forks source link

Schema evolution handling #39

Open guersam opened 10 years ago

guersam commented 10 years ago

Hi,

Scala Pickling looks highly promising so that I want to migrate an existing Eventsourced application which uses default Java serialization for event storage.

Currently the biggest concern other than #27 is schema versioning. The application requirement changes as time goes by, so eventually I'll need to add/rename/remove some fields from event messages, and then compatibility with previous versions of message will be an issue.

Is there any plan or recommendation about this?

crumbpicker commented 10 years ago

+1 Maybe Option could be an option ? I hope there will be support for this.

guersam commented 10 years ago

Probably already checked, but FYI just in case: Schema evolution in Avro, Protocol Buffers and Thrift

phaller commented 10 years ago

At the moment it is necessary to create custom picklers to deal with schema evolution (for example, a custom pickler could unpickle older versions of a message, but create instances of newer versions).

However, it is possible to create new pickle formats which are more evolution-friendly, for example, by including something akin to Protobuf's field tags (the blog post linked above explains well how these tags help with schema evolution). Adding an evolution-friendly variant of the binary pickle format would be very useful, indeed.

guersam commented 10 years ago

Thanks for your advice, @phaller. At least, default parameter support will provide some basic compatibility out of the box, but it seems difficult without additional annotation until SI-5920 is resolved.

I decided to go with Protobuf for now but will keep eyes on Pickler looking for what I can contribute.

adamdecaf commented 9 years ago

Since 0.10.0 came out I was looking at pickling for use to persist messages and was thinking about how schema evolutions would happen. Sadly this test I wrote didn't pass, but I'm mentioning it here to see if now that some of those PR's in the SI have been merged if it's likely for pickling to have support for handling default methods. It looks like it'll be fixed in scalac in 2.11.6?

https://github.com/adamdecaf/pickling-migration-test/blob/master/src/main/scala/Add.scala

Deleting fields works though!

havocp commented 9 years ago

In https://github.com/sbt/serialization we support Option fields that can be missing.

pjan commented 9 years ago

It would be totally awesome if schema evolution came about. It currently is the thing keeping us back from using pickling. @adamdecaf's outlined usecases pretty much sum up the hoped-for functionality for now.

protobuf/avro/thrift was mentioned before, but it is just not cutting it in a pure scala-based environment due to the lack of support for e.g. polymorphism, drastically cutting down the flexibility that scala offers.

So, very keen to get an update on this :smile: