Serializing empty lists and deserializing empty files

[GoogleCodeExporter]

Please include an e-mail address if this might need a dialogue!
==============

What steps will reproduce the problem?
1. Create a List<T> that's serializable
2. Serialize the empty List<T> of any type to a file
3. Deserialize the empty list from the file

What is the expected output? What do you see instead?
Expected: An empty list
Seen: Null

Repeating this process eventually causes an InvalidOperation exception.
After the InvalidOperation exception has occured, all files with data in
them fall over with the error that only DataContract members, etc can be
deserialized, or a TypeException.

What version of the product are you using? On what operating system?
Latest available. Windows Mobile 6.0.

Please provide any additional information below.

Original issue reported on code.google.com by dandreds...@gmail.com on 27 May 2009 at 10:29

[GoogleCodeExporter]

The "null" behaviour is a side-effect of the wire format that is defined by 
google. 
Essentially, there is no way of distinguishing between a null list and an empty 
non-
null list without breaking compatibility with the data format. So as an 
arbitrary 
choice, it returns null. If this is a member of a class, there are some tricks 
that 
can be employed to give more expected behaviour.

Re the InvalidOperationException; what is the message? I am aware of a few 
limitations inside CF, and there is work in progress to avoid these issues (but 
by 
pretty much a re-write to a different approach). Du to the complexity etc this 
refactor is taking a bit of time (it currently passes about 80% of the 
serialization 
unit tests - I haven't started deserialization yet). It may be that the 
refactored 
code avoids this issue.

Original comment by marc.gravell on 27 May 2009 at 11:24

[GoogleCodeExporter]

The message returned is "Only data-contract classes (and lists/arrays of such) 
can be
processed (error processing List`1)"

I created a work-around by inserting a if (FileStream.Length > 0) within my 
stream
creation. This doesn't seem to completely the fix the issue. Since I added 
TripleDES
Encryption I get the same thing when deserializing from a CryptoStream 
(sometimes,
not always!).

This error seems to happen very inconsistently. It will sometimes work and 
sometimes
not, even with the same files.

Original comment by dandreds...@gmail.com on 28 May 2009 at 4:18

[GoogleCodeExporter]

Please find my code attached

Original comment by dandreds...@gmail.com on 28 May 2009 at 4:33

Attachments:

• Bugged.txt

[GoogleCodeExporter]

I will take a look. For interest, there is now (via the new "packed" encoding 
in the 
update wire format) a way to distinguish empty vs null lists directly, but it 
only 
works for values like "int" (not classes). The changes to support this in 
protobuf-net 
should be committed in the next day or so.

Original comment by marc.gravell on 28 May 2009 at 7:20

[GoogleCodeExporter]

Just to note with that code I supplied, that's in a base class that's inherited 
by
all of my data providers. It seems that it only errors after I've loaded a 
certain
number of times, i.e. dependant on the direction with which I move through the
system. The load event is where the error occurs.

Basically, there is a register, an assignment screen and a customer screen. If 
I go
straight to the customer aspect, which loads the customers file, that works 
fine.
Then I go to the register which loads another 3/4 files, one of those files 
usually
conks out. Then if I restart and access the register first - works fine - 
customer
screen then has the same issue.

It's looking a memory issue to me, but I've not looked into the source code of
ProtoBuf.Net to see what's actually happening when the error occurs. I don't 
have
this problem when the files are generated on my webservice, which again points 
to a
mobile memory error. This error has only started to occur reguarly since I 
added the
encryption streams into it. If I remove the encryption streams and just write 
to a
memorystream on save there are no issues.

Let me know if I can be of any more help.

Original comment by dandreds...@gmail.com on 29 May 2009 at 6:39

[GoogleCodeExporter]

Can you let me know what the exact exception is? I'm aware of, and in the 
process of 
addressing, some internal limitations in CF (i.e. the CF engine itself, not the 
protobuf-net code) which can cause things like "MissingMethodException" 
occasionally.

Original comment by marc.gravell on 29 May 2009 at 7:41

[GoogleCodeExporter]

As per comment 2, it's "Only data-contract classes (and lists/arrays of such) 
can be
processed (error processing List`1)"

Original comment by dandreds...@gmail.com on 29 May 2009 at 7:49

[GoogleCodeExporter]

InvalidOperationException is the exact exception

Original comment by dandreds...@gmail.com on 1 Jun 2009 at 4:30

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I downloaded the source code and stepped through until the error occured (it 
was on
about the 10th load of data [each from different files/tables]). The actual 
error
inside your source code is MissingMethodException, this is the stack trace:

at ProtoBuf.Property.PropertyFactory.CreatePassThru()
at ProtoBuf.Property.PropertyList`3.OnBeforeInit()
at ProtoBuf.Property.Property`2.OnBeforeInit()
at ProtoBuf.Property.Property`1.InitPrivate()
at ProtoBuf.Property.Property`1.Init()
at ProtoBuf.Property.PropertyFactory.CreatePassThru()
at ProtoBuf.SerializerProxy`1..cctor()
at ProtoBuf.Serializer.Deserialize()
at ProtoBuf.Serializer.Deserialize()
at [Removed].Dal.DalProvider`1.Load()
at [Removed].Dal.DalProvider`1.set_FilePath()
at [Removed].Dal.DalProvider`1..ctor()
at [Removed].Dal.PaymentEntityProvider..ctor()
at [Removed].Session.BizSession.get_TraderPayment()
at [Removed].Helpers.TraderHelper.GetTraderAccount()
at [Removed].fUnitCollections.PopulateCollectionsList()
at [Removed].fUnitCollections.Initialise()
at [Removed].fMarketMenu.ShowCollections()
at [Removed].fMarketMenu.btnCollections_Click()
at System.Windows.Forms.Control.OnClick()
at System.Windows.Forms.Control.WnProc()
at System.Windows.Forms.ContainerControl.WnProc()
at System.Windows.Forms.Control._InternalWnProc()
at Microsoft.AGL.Forms.EVL.EnterModalDialog()
at System.Windows.Forms.Form.ShowDialog()
at [Removed].Pda.CodeBank.tmsDialogResult.GetValidDialogResult()
at [Removed].Pda.CodeBank.tmsDialogResult.GetYesOrCancel()
at [Removed].fMainMenu.StartMarket()
at [Removed].fMainMenu.btnMarketSelect_Click()
at System.Windows.Forms.Control.OnClick()
at System.Windows.Forms.Control.WnProc()
at System.Windows.Forms.ContainerControl.WnProc()
at System.Windows.Forms.Control._InternalWnProc()
at Microsoft.AGL.Forms.EVL.EnterMainLoop()
at System.Windows.Forms.Application.Run()
at [Removed].Program.Main()

The error occurs inside: static SerializerProxy() in the else statement.

Original comment by dandreds...@gmail.com on 2 Jun 2009 at 8:08

[GoogleCodeExporter]

OK; the MissingMethodException issue is the problem I am currently working on 
in the 
"big refactor"... This is hitting an internal CF limitation on generics, and 
there is 
no simple way of fixing it in the current codebase. This work is in progress, 
but I 
expect it to take another month or so (it is not simple).

See: 
http://marcgravell.blogspot.com/2009/03/compact-framework-woes-revisted.html

Original comment by marc.gravell on 2 Jun 2009 at 8:22

[GoogleCodeExporter]

Added to front page too

Original comment by marc.gravell on 2 Jun 2009 at 8:24

[GoogleCodeExporter]

Thanks Marc, however I don't understand why the error only occurs when I start 
using
the CryptoStream? If I remove the cryptostream I can happily use the 
application with
no problems.

Original comment by dandreds...@gmail.com on 2 Jun 2009 at 8:26

[GoogleCodeExporter]

My assumption would be that the generics (which is what underlies the 
MissingMethodException) and the crypto-stream are competing for one of the 
heavily-
constrained resources in CF... there's room for one but not both. Perhaps. Very 
hard to 
give a definitive answer.

Original comment by marc.gravell on 2 Jun 2009 at 8:29

[GoogleCodeExporter]

Sorry, just to correct myself... it works for longer when not using the 
cryptostream,
i.e. with the Crypto I can open 4/5 tables and without I can open about 10 
before the
same error happens.

It is as though something is holding onto resources somewhere and not releasing 
them
properly? It's definitely not in my code.

Are there any work arounds I can do? I was meant to be deploying tomorrow! 
Woops.

Original comment by dandreds...@gmail.com on 2 Jun 2009 at 9:17

[GoogleCodeExporter]

Marc, 

Am I understanding correctly that ProtoBuf-net builds up a cache of data 
structures,
etc? Is there any way to clear these structures and would it cause any side 
effects?

Original comment by dandreds...@gmail.com on 2 Jun 2009 at 10:32

[GoogleCodeExporter]

You could edit the source to set Serializer<T>'s "readProps" and "writeProps" 
to null 
(and possibly "subclasses" and "callbacks", but you'd need to do that for 
multiple T 
(not just the root type). Unfortunately, the issue is related to the number of 
generic types in use, so it is likely to come back... I'm refactoring it to 
make 
almost no use of generics, which should avoid this issue.

In the refactored model the entire object graph can be released trivially (or 
you can 
have multiple different graphs for the same types).

Unfortunately the refactored version is nowhere near production use - it only 
passes 
about 75% of the tests, and I haven't even *started* on *de*serialization.

Original comment by marc.gravell on 2 Jun 2009 at 12:21

• Added labels: YoucouldeditthesourcetosetSerializerTsreadPropsandwritePropstonullbutyoudneedtodothatformultipleTnotjusttheroot.Unfortunatelytheissueisrelatedtothenumberofgenerictypesinusesoitislikelytocomeback...Imrefactoringittomakealmostnouseofgenericswhichshouldavoidthisissue.Unfortunatelytherefactoredversionisnowherenearproductionuse-itonlypassesabout75ofthetestsandIhaventevenstartedondeserialization.

[GoogleCodeExporter]

Ok, I've migrated back to XmlSerializer for the time being, please keep us 
posted.

Original comment by dandreds...@gmail.com on 2 Jun 2009 at 2:14

[GoogleCodeExporter]

Have the same issue, will it be fixed or because of wire format I should do 
workaround?

    [DataContract]
        public class Data
        {
            [DataMember(Order = 1)]
            public List<string> List { get; set; }
        }

        [Test]
        public void EmptyListIsEmpty()
        {
            var ser = new DataContractSerializer(typeof(Data));
            var stream = new MemoryStream();
            ser.WriteObject(stream, new Data{List = new List<string>()});
            stream.Position = 0;
            var data = ser.ReadObject(stream) as Data;
            Assert.IsTrue(data.List.Count == 0);
        }
        [Test]//FAILS
        public void EmptyListIsEmpty()
        {
            var ser = new ProtobufObjectSerializer();
            var stream = new MemoryStream();
            ser.WriteObject(stream, new Data { List = new List<string>() });
            stream.Position = 0;
            var data = ser.ReadObject(stream, typeof(Data)) as Data;
            Assert.IsTrue(data.List.Count == 0);
        }

Original comment by asd.and....@gmail.com on 19 Mar 2013 at 11:49

[GoogleCodeExporter]

ProtobufObjectSerializer is just wrap to make protobuf-net interface more like 
DataContractSerializer. Sorry for bad copy paste...

Original comment by asd.and....@gmail.com on 19 Mar 2013 at 11:50

[GoogleCodeExporter]

It is unclear to me what I would change here; there is simply no conceptual 
difference here in terms of the raw spec (or more accurately: there is no 
concept of null in protobuf).

Original comment by marc.gravell on 19 Mar 2013 at 12:16

[GoogleCodeExporter]

Ok. Seems to be the answer, but to be clear.
My question is - we had DataContractsSerializer in WCF and it worked.
Used protobuf-net and got different behavior regarding empty lists.
Our code fails now.
So end solution is to fix our code because protobuf spec prevents fixing it in 
library?

Original comment by asd.and....@gmail.com on 19 Mar 2013 at 12:30

[GoogleCodeExporter]

May be some wiki page needed like "Migrating from DataContractsSerializer to 
protobuf" mentioning such issues and closing this bug as no fix.
Other content for wiki will be - put Order on all elements or got null.

Original comment by asd.and....@gmail.com on 19 Mar 2013 at 12:32