Serializing structs / immutable classes

[GoogleCodeExporter]

       What's the best way for me to get this to work with protobuf-net? Is 
there a way for me to inject a serializer proxy thing or something? (I was 
so sure it worked at one point, but obviously not.)

Thank again,
-Michael

   [DataContract(Namespace = FileDbSerializer.XmlDbNamespace)]
   public struct KeyPair<TKey1, TKey2>
   {
       public KeyPair(TKey1 k1, TKey2 k2)
           : this() {
           Key1 = k1;
           Key2 = k2;
       }
       // Stupid tuple class for datacontract
       [DataMember]
       public TKey1 Key1 { get;  internal set; }
       [DataMember]
       public TKey2 Key2 { get;  internal set; }

       public override string ToString() {
           return Key1.ToString() + ", " + Key2.ToString();
       }
   }

Original issue reported on code.google.com by marc.gravell on 12 Oct 2008 at 7:44

[GoogleCodeExporter]

Original comment by marc.gravell on 26 Jun 2009 at 7:27

• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

This is quite a blocker for me. I have lots of structs (Rect, Point, and similar
classes) which mostly contain ints. The only way I've figured out to serialize 
them
with protobuf is to have ToString() & Parse() combination, which is quite bad.

Original comment by to...@iki.fi on 16 Nov 2009 at 5:09

[GoogleCodeExporter]

I don't have a good answer for this with the current implementation. This is 
one of the 
things I have on the list for some refactoring work I have planned, but the 
problem is 
finding time (it is a major change).

Original comment by marc.gravell on 16 Nov 2009 at 8:54

[GoogleCodeExporter]

We also have the same issue. 

--- Use-case --- 
We're storing some 1.000.000.000 instances of a very small structs on a server, 
which
needs to be persisted and loaded at startup. The memory overhead of swithcing to
objects would be about 24 GB (!) (cf.
http://www.simple-talk.com/dotnet/.net-framework/object-overhead-the-hidden-.net
-memory--allocation-cost/).
Also, I assume we'd end up with decreased performance due to object allocation, 
GC,
non-sequential order in memory (cache misses), etc? I'd love to switch to 
Protocol
Buffers all over the product, but here it doesn't seem to be practical?

Marc, I rather recently discovered the project. Are you pretty much working 
alone on
it or is it possible to help out in some way?

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 8:03

[GoogleCodeExporter]

@andersgsjogren - then you may be glad to hear that v2 supports (mutable) 
structs. Will 
that help at all? It is indeed a sole effort at the moment, but I'm open to 
offers of 
help.

Original comment by marc.gravell on 22 Apr 2010 at 9:00

• Changed state: Started

[GoogleCodeExporter]

Also - I *hope* to think of some clever way of enabling immutable structs, 
since let's 
face it: mutable structs are evil. At the moment I'm looking at solving this a: 
for 
XNA, and b: because it fell out quite easily during the re-design for v2. I 
need to 
apply a bit of thought, but I very much hope to have a "immutable" solution, 
although 
it might involve (for example) having a mutable twin-strict and bidi-conversion 
operators.

Original comment by marc.gravell on 22 Apr 2010 at 9:02

[GoogleCodeExporter]

(of course, the publicly immutable approach, as illustrated in the first post 
here, is 
reasonable; the problem is that I can do this for the "in memory" compile, but 
I can't 
use this approach for the "pre-compile to a dll" model)

Original comment by marc.gravell on 22 Apr 2010 at 9:11

[GoogleCodeExporter]

Original comment by marc.gravell on 22 Apr 2010 at 9:17

• Added labels: Milestone-Release2.0

[GoogleCodeExporter]

Well, looking a bit more at the implementation, it seems to be possible to work
around it, using a kind of Surrogate pattern and the Merge method for avoiding 
object
allocation. Can this be done better?

public struct Point
{
    private const PrefixStyle Style = PrefixStyle.Base128;

    public int X;
    public int Y;
    public Point(int x, int y) { X = x; Y = y; }
    private Point(Surrogate s) { X = s.x; Y = s.y; }

    [DataContract]
    private class Surrogate
    {
        public void GetObjectData(Point p) { x = p.X; y = p.Y; }
        [DataMember(Order = 1)]
        public int x;
        [DataMember(Order = 2)]
        public int y;
    }

    public static void SerializeMany(IEnumerable<Point> points, Stream stream)
    {
        var reusedSurrogate = new Surrogate();
        foreach (var point in points)
        {
            reusedSurrogate.GetObjectData(point);
            Serializer.SerializeWithLengthPrefix(stream, reusedSurrogate, Style);
        }
    }

    public static IEnumerable<Point> DeserializeMany(Stream stream)
    {
        var reusedSurrogate = new Surrogate();
        while(true)
        {
            try
            {
                Serializer.MergeWithLengthPrefix(stream, reusedSurrogate, Style);   

            }
            catch(EndOfStreamException) //How should this be done?
            {
                yield break;
            }
            yield return new Point(reusedSurrogate);
        }
    }
}

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 9:39

[GoogleCodeExporter]

[My last comment (#9) was ignorant of comments 5 and below.]

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 9:40

[GoogleCodeExporter]

Would "but I can't use this approach for the "pre-compile to a dll" model" mean 
that
you can't use reflection in other words? What about emitting code using
PropertyInfo/MethodInfo (or the like) of private (property) setters that are 
being
found using reflection at pre-compile-time.

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 9:44

[GoogleCodeExporter]

> then you may be glad to hear that v2 supports (mutable) structs. Will that 
help at all?
Yes, but immutable ones would be even better (as you say). Could it be an 
option to
support immutable ones for the non-precompiled version?

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 9:46

[GoogleCodeExporter]

For mutable structs (or where the field is mutable but hidden behind a private 
`set`) 
it should work in "v2", although I can't guarantee it "today" (I'll the 
scenario as a 
specific test, though). The surrogate is what I'm thinking for long-term / 
immutable - 
i.e. have your immutable struct that has conversion operators defined to a 
surrogate 
*mutable* struct; I call the operator when fetching, work on the mutable 
struct, and 
then call the reverse operator prior to storing. That work?

Original comment by marc.gravell on 22 Apr 2010 at 9:47

[GoogleCodeExporter]

A side note: I guess my comment [#9] was an example of "mutable twin-strict and
bidi-conversion operators.", only that it was not a twin struct but a twin 
class?

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 9:47

[GoogleCodeExporter]

> I call the operator when fetching, work on the mutable struct, and then call 
the
reverse operator prior to storing. That work?
It's certainly better than not at all, but not as good as a solution without the
twin/sorrogate... ;-)

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 9:50

[GoogleCodeExporter]

So, I'm uniformed on the subject, but why is it that you cannot include private
setters in the pre-compiled code?

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 9:51

[GoogleCodeExporter]

The problem with immutable is that I'd need to do ctor-mapping, and keep the 
values 
locally (which doesn't play nicely against how the engine works). A surrogate 
would 
be simpler. For info, the problem with compiled **to an external dll** is field 
access:

        [Test, ExpectedException(typeof(FieldAccessException))]
        public void FullyCompileWithPrivateField_KnownToFail()
        {
            var model = BuildModel();
            Point point = new Point(26, 13);
            ClonePoint(model.Compile(), point, "Compile"); // clones and compares
        }

However, it works fine if just compiled in-memory, as we can assert stronger 
access:

        [Test]
        public void RoundTripPoint()
        {
            Point point = new Point(26, 13);
            var model = BuildModel();

            ClonePoint(model, point, "Runtime");

            model.CompileInPlace();
            ClonePoint(model, point, "CompileInPlace");
        }

Hence why *if you need it as a pre-compiled dll*, a surrogate is my simplest 
option. 
Of course, you only *benefit* from pre-compiled for lightweight frameworks like 
CF, 
MonoTouch, etc. For full-fat .NET the in-memory compile actually makes a faster 
serializer.

Original comment by marc.gravell on 22 Apr 2010 at 10:01

[GoogleCodeExporter]

Because the CLI is rude enough to check that you haven't violated the 
accessibility 
rules; if you have, it throws an exception. However, if you have enough 
permission 
(/trust) to write IL directly in-memory, then `DynamicMethod` allows you to say 
"oh, 
and pretend that this method belongs to type X", providing you the same access 
to 
non-public members that a method of X would have. Meaning: it can access 
private 
fields and call private methods.

I *believe* (although I haven't checked) that this is exactly the same as how 
`DataContractSerializer` does the equivalent thing.

Original comment by marc.gravell on 22 Apr 2010 at 10:19

[GoogleCodeExporter]

Thanks for the info! If I understand correctly, the point example above (but 
with X
and Y being private set properties) would then work without the need for 
Surrogates
in v2?

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 11:31

[GoogleCodeExporter]

On #19: For the full CLR that is.

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 11:34

[GoogleCodeExporter]

Let me clarify; it won't work **at all** in v1.

v2 offers 2 different types of compilation; in-memory (type-by-type), and pre-
compiled. The type-by-type serialization *will* work with private fields and 
private 
setters. The pre-compiled entire model approach (which, confusingly, can be 
either 
in-memory or to a dll) will *not* work with private members. But the good news 
is 
(tada) surrogates now work, as of 5 minutes ago. I haven't done a "commit", and 
there 
isn't an attribute for expressing this yet (you can do it with the new 
object-model, 
though), but *it works*.

Original comment by marc.gravell on 22 Apr 2010 at 12:54

[GoogleCodeExporter]

I promise I'll do a better job of explaining this when I release it, but right 
now 
you're seeing it "live" so to speak ;-p

Original comment by marc.gravell on 22 Apr 2010 at 12:55

[GoogleCodeExporter]

Thx for the clarification (and the good work). Looking forward to v2. Any 
estimates
on how much work is left before it's stable enough for production use?

Original comment by andersgs...@gmail.com on 22 Apr 2010 at 7:29

[GoogleCodeExporter]

Hi, I'm running into this same issue.
How bout leveraging ISerializable for serialization of immutable structs? You 
could use? ISerializable.GetObjectData for serializing the objects and use the 
.ctor(SerializationInfo, StreamingContext) for deserializing/constructing the 
structs? I'm not very familiar with the inner workings of protobuf-net but on 
the surface this looks like workable solution, don't you think?
In fact I think this is a scenario that could be supported in general for types 
that do implement ISerializable but don't have the ProtoContract/DataContract 
attribute markers. There are many such structs in the BCL.

Original comment by jimitndi...@gmail.com on 4 Oct 2010 at 2:43

[GoogleCodeExporter]

v2 already has better support for this (including structs).

ISerializable is problematic in that a: it isn't avalable for all supported 
platforms, and b: there is no means to map such data to a valid proto stream 
(except as a non-portable blob)

Let me know I'd you want a v2 example, but note: not quite released yet.

Original comment by marc.gravell on 4 Oct 2010 at 3:09

[GoogleCodeExporter]

Is the feature illustrated in #17 incorporated in the current trunk version? 
I'm using in-memory compilation in v1 (with the full .net stack). If the 
current trunk supports this with in-memory compilation I'd definitely take a 
look at it.

Original comment by jimitndi...@gmail.com on 5 Oct 2010 at 10:26

[GoogleCodeExporter]

@jimitndiaye - yes, that is the v2 trunk running against an immutable struct, 
operating against the fields:

    [ProtoMember(1)] private readonly int x;
    [ProtoMember(2)] private readonly int y;

In the unit test the struct is decorated attributes, but the same would work 
using a runtime type-model (if the struct isn't under your control).

Original comment by marc.gravell on 5 Oct 2010 at 10:39

[GoogleCodeExporter]

Could you elaborate on "but the same would work using a runtime type-model (if 
the struct isn't under your control)". How would you use a runtime type-model 
to serialize a struct not under your control?

Original comment by jimitndi...@gmail.com on 6 Oct 2010 at 10:45

[GoogleCodeExporter]

Original comment by marc.gravell on 13 Jun 2011 at 9:06

• Changed state: Fixed