[BUG]: Unable to deserialize class with Array

[smolesen]

Library Version

4.22.0

OS

Windows

OS Architecture

64 bit

How to reproduce?

1. Create a class with an array of int or double
2. Serilize the class into a memorystream using: await ParquetSerializer.SerializeAsync(new List { new EmptyArrayClass { Name = "SomeName", Values = new [] { 0 } } }, stream);
3. Update position in memorystream to 0
4. Deserialize stream into object, this will throw a 'Value cannot ne null'

Failing test

public class EmptyArrayClass
   {
       public string Name { get; set; }
       public int[] Values { get; set; }
   }

   [Test]
   public async Task Deserialize_Empty_Array()
   {
       var stream = new MemoryStream();
       await ParquetSerializer.SerializeAsync(new List<EmptyArrayClass> { new EmptyArrayClass { Name = "SomeName", Values = new [] { 0 } } }, stream);
       stream.Position = 0;
       var result = ParquetSerializer.DeserializeAsync<EmptyArrayClass>(stream);
   }

[aloneguid]

I appreciate your thorough report, it makes the bug fixing process much easier. I'm working on it right now and I'll keep you updated on the progress.

[aloneguid]

@smolesen as a workaround you can replace int[] with List<int> as arrays are not supported by serializer yet.

[smolesen]

Yeah, I tried that, it just gives me a lot of problems in the rest of the code.... of course I could deserialize as List and copy it to another object with int[] BTW: It used to work ... in version: 3.8.6 /Søren

[aloneguid]

@smolesen in 3.8.6 it was extremely inefficient and only worked for root-level arrays :(

[smolesen]

Aha I see...

I have a lot of data written with 3.8.6, do you think I'll be able to deserialize it, if I use List or will I have to convert it first?

[aloneguid]

Yes, but depending on how it was written you might need to mark arrays as legacy (if they are), see https://aloneguid.github.io/parquet-dotnet/serialisation.html#legacy-repeatable-legacy-arrays

[aloneguid]

@smolesen the complexity of deserializing into array is we don't know beforehand the size of the array, so deserialize cannot pre-allocate correct size. V3 was only operating on simple properties and was deserializing arrays into list of lists first, and then converting to arrays. So it will be more efficient to use List<T> in the beginning.

The only workaround in deserializer i'm thinking of is allocating empty array, then resizing it every time a new element comes in, but that might quickly put a lot of pressure on GC.

[aloneguid]

so it's something like this for each element:

[smolesen]

Ok, think I found a workable way to use List<> instead, so I'll stop using array.

Thanks for looking into it though....

[aloneguid]

Latest version also supports arrays of primitives btw if you still need it.

[smolesen]

Great and thanks....