Add ParquetReader with Cardinality information

illinoisdata / DeepOLA

7 stars 5 forks source link

Add ParquetReader with Cardinality information #142

Closed nikhil96sher closed 1 year ago

nikhil96sher commented 1 year ago

Add ParquetReader with Cardinality.
Modified the structure of data directories (added an extra tbl at the end) to store multiple formats.
Added script to convert to parquet.

nikhil96sher commented 1 year ago

The ParquetReader process_stream is a copy of the StreamProcessor of CSVReader. I am not sure of a better solution to avoid duplicate code (right now it conflicts with other MessageProcessor implementation).