To avoid defining a parquet schema for each struct and need to update this schema every time a new field is added, the method create_parquet_schema_and_data was introduced. This function takes a list of serializable items, in this case ExpresisonResult structs, and generates a Parquet Schema and a vector of ArrayRef that represents the data for each field in the schema. The schema and data can then be used to write to Parquet files or perform other data processing operations.
Summary
Adds parquet serialization capability
Technical details
To avoid defining a parquet schema for each struct and need to update this schema every time a new field is added, the method
create_parquet_schema_and_datawas introduced. This function takes a list of serializable items, in this caseExpresisonResultstructs, and generates a ParquetSchemaand a vector ofArrayRefthat represents the data for each field in the schema. The schema and data can then be used to write to Parquet files or perform other data processing operations.PR Checklist