Problem with limiting API for defining custom picklers

[Horusiath]

I've got following problem: I need to integrate with 3rd party library, which produces objects, that are not serializable, unless I will define custom way of pickling/unpickling for them. Those objects can be composed in fields of other objects, collections etc. I cannot annotate them with [<CustomPickler>], but I can recognize them by interface, they implement. Therefore the only possible option here seems to be defining custom pickler factory for that interface and use FsPickler.RegisterPicklerFactory on it. This however introduces some other problems:

• This is global function, while for making my custom serialization process possible I need to pass a reference to dedicated object that will supply pickler with necessary data. It would be perfect, if I could register factory on particular FsPicklerSerializer instance level.
• Another problem is that I can define pickler factory for a particular interface, but it needs to be actually applied for all implementations of that interface. Current design don't allow to do so without using reflection - as FsPickler.RegisterPicklerFactory works on generic types only, and it's not interface-wise. Also this object can be nested inside another objects, so pickler needs to be composable as well.

Any clues how to solve these problems? Thanks in advance.

[eiriktsarpalis]

Hi Bartosz.

Regarding your points:

• In general, picklers are intended as general-purpose serializers at the type definition level and should not encapsulate any context-specific state. For this and many other reasons picklers are generated globally. Is it possible to solve your problem by passing state with StreamingContext?
• Try using the Pickler.FromPrimitives method with the useWithSubtypes flag set to true. This will force that particular pickler implementation for any subtype of the given interface.

In any case, could you give a short code sample of what you are trying to do?

[0x53A]

Hi Eirik,

I am in a similar situation to Horusiath. (In my case I want to serialize Akka.Net objects, e.g. IActorRef).

If you have some time, could you please take a look at https://github.com/mbraceproject/FsPickler/pull/82?

This should enable the scenario Bartosz and I require, where a factory can be registered to a custom non-global context, while also keeping the default global behaviour.

[eiriktsarpalis]

Hi Lukas,

Apologies for the delayed response. The singleton pickler repository was deliberately introduced in later versions of the library. This happens because generating picklers is expensive and involves a great deal of code generation. In a typical application (like MBrace) a single pickler cache would take up to 4MB of memory. We want to avoid people misusing the library by generating a new set of picklers every time they serialize or deserialize something.

[eiriktsarpalis]

That said, I can think of few ways the library could make pickler caches cheaper, but it doesn't completely eradicate the problem and requires significant refactorings and breaking changes to the API.

[0x53A]

But how would it hurt to keep the current behavior by default (the singleton instance) while at the same time enabling independent parallel instances?

This would be backwards-compatible, both from a (binary-)api point and from a performance point.

It would only (possibly) degrade performance, when I actually spin up multiple containers, but when that happens, it is my explicit choice.

The current workarounds are even worse, because the only way to get isolation at the moment is by spinning up a new AppDomain.

[eiriktsarpalis]

We had very real space leak issues because of this. People are inclined to create a new serializer instance on each serializer/deserialization cycle even though the API in FsPickler is thread safe.

[0x53A]

@eiriktsarpalis So this is just a case of not giving the user enough rope to hang themselves with?

You could either mark the new API as obsolete, or add them to an Internal namespace to warn the users, but I think not having it is a big issue because it just hard-blocks certain scenarios.

In my case I want a context per Akka.net ActorSystem. There is (mostly) only one or two systems over the whole lifetime of the application.

[dsyme]

Closing this old discussion/question