Questions about "_flags"

[GoodGuyJim]

Hey there, just had two quick questions about how certain things in my Pickled JSON end up as an object that looks like:

{ "_flags": "cached", "id": 45 }

Is that some form of JSON compression? If so, is there any way to return the uncompressed JSON without attempting to convert it to the type that was used to make it in the first place?

Thanks!

[eiriktsarpalis]

Hi Jim,

This representation is not particular to JSON, it is used by FsPickler independently of format. Nodes in an object graph that are referentially equal will never be serialized twice. This is to reduce serialization size and to properly treat cyclic objects. Unfortunately, this behaviour cannot be changed.

In terms of expanding the JSON pickle, I'm afraid you're on your own here. You would have to write custom code that manipulates the JSON tree.

I'd be interested though to find out why you would want to do this.

[GoodGuyJim]

I'm just going to say all this and you'll either understand me or you won't, in which case feel free to just smile and nod and then mark this issue closed :-)

We're using FsPickler to serialize and then save events in an Event Store on a CQRS system. We're trying to add event versioning where we use the F# Json type provider to detect where events have been changed and update them appropriately. So we've been looking a lot at the actual JSON that we've been saving to the database and we've noticed that the compression sort of "tricks" the type provider, since sometimes things are their normal shape and sometimes they are that cached object. So we were just wondering if we could get the "uncompressed" JSON out and work with that.

Thanks for the reply!

[eiriktsarpalis]

Ok, that's pretty interesting. If it helps, there are a few ways with which you can reduce the chance of identical objects appearing in an object graph. Given that you are using EventStore I would imagine that you are persisting small, immutable objects. So here's a few points:

• Do you serialize each of the events independently or do you serialize a whole collection of them at once? This can make a big difference since you are containing serialization state to smaller object graphs. In case of the same event (up to reference equality) appearing in the list you would avoid it being "cached".
• Do your events capture singleton objects? The usual suspects are instances of System.Type or user-defined entities.
• Does your domain modelling rely on subtyping and inheritance? If your persisted event type is an abstract class that is being inherited by many concrete implementations then FsPickler will compensate by serializing reflection data, which may often be "cached". A common remedy is to use F# discriminated unions instead.

Finally, here's an example that illustrates when and how this caching behaviour kicks in:

let jsp = FsPickler.CreateJson(indent = false, omitHeader = true)

let v1 = let x = [1..10] in x,x
let v2 = [1..10], [1..10]

jsp.PickleToString v1
jsp.PickleToString v2

The two values are structurally identical, however they will produce different serializations, the first yielding

{"Item1":[1,2,3,4,5,6,7,8,9,10],"Item2":{"_flags":"cached","id":2}}

and the second

{"Item1":[1,2,3,4,5,6,7,8,9,10],"Item2":[1,2,3,4,5,6,7,8,9,10]}

I realize that you probably need to work with data that has already been serialized, so these points might not make much of a difference to you.

[GoodGuyJim]

Thanks for the detailed reply!

Just so we're clear, we're actually using our own event sourcing solution at the moment, we started off about a year ago using NEventStore but we didn't love how CommonDomain forced us to structure our domain; and we were going to use EventStore but that didn't pan out for us (for reasons that aren't really important).

We serialize each event and save it independently. Our domain ONLY uses discriminated unions, we aren't using any inheritance in anything that was created from scratch in F# (we have some legacy C# stuff that uses interfaces, but they don't interop too much).

Yeah our data is already saved so we don't have any say in it, but thanks for the examples, it's interesting to see how it works.