Expando Objects

[jtgooding42]

Questions, if this is already somewhere I apologize, I looked through the documentation and google etc.

1) You had posted on a guy who made a method for DataTable to Parquet asking to add his code and then you left a post, he granted it and you replied that you were adding it. This was pre-async days does this code still exist? I really want DataReader but can get around it by loading X rows into DataTable then add row group clear table etc. but the logic for DataTable doesn't appear to exist... 2) In the meantime, I thought a more lightweight version might be to do it with an Expando Object or more accurately List since each parquet file I am creating is of unknown fields and field types until I query it. Doing this I get an error when using it 'at least one field is required (parameter 'fields') now it's totally possible I am doing this completely wrong...

Code Sample: await ParquetSerializer.SerializeAsync(Rows, OutputFilePath, options); where rows is defined as a List.

I am trying to do this in batches to avoid the row api, but thought I would ask is there a way of doing either of these before I go re-inventing the wheel etc. Thanks

[aloneguid]

Hi. Your best bet is to use the low level API as you can build schema and data in runtime. The issue with System.Data is that it doesn't support complex types so there is no way to serialise even something simple like structs.

[jtgooding42]

Thanks, I realize that Parquet has an absurd amount of different ways to store data but from my experience, well so far on about 200 systems not one of those has ever been used. With the popularity of Snowflake, Databricks, Fabric a popular real world usage today is much more traditional types you might use than structures, embedded arrays etc.

It's just like SQL Server yeah you can create custom types, CLR types etc. but they tend to perform poorly and cause issues somewhere through some tool. I'm not saying drop that kind of support by any means but not supporting many mainstream usages because it doesn't work with one feature just means a lot of people writing the same broiler plate code.

Love the product and I have already built my version of the above code since yesterday as I assumed this would be your answer, I may even implement an IDataReader interface for the parquet reader today. The advantage of this library is .NET over java but it's not embracing the data aspect of working in .NET to implement normal programming paradigms.

I am totally fine with the serialize over data table, data reader for writing throwing an exception saying datatype not supported when it comes to those extra features as I know I won't see them in my workloads.

Thanks again for a great product, just my 2c which doesn't count as much :)

[aloneguid]

@jtgooding42 totally agree here, my use cases are also most of the times consist of flat data only. I have followed the similar approach when implementing DataFrame integration - just using primitive types. Would really appreciate you adding System.Data integration and happy to merge it into master here!

[jtgooding42]

I have some sample boiler plate code for datareader to parquet. I tested the table/row api, the new dictionary (untyped) and a home grown low level api, and the row api and the low level produce nearly same timings with low level a bit faster, the untyped was almost twice as long, I would love for ya to take a look and see if I did anything obviously wrong in each of the 3, or recommend perf improvements, I'm fine with the code being incorporated but would guess that it would require a fair bit of rework to be api friendly. Lastly the new untyped api takes a dictionary<string, object>, not sure how to pass nulls in on that shouldn't it be dictionary<string, object?>.