search

mukunku / ParquetViewer

Simple Windows desktop application for viewing & querying Apache Parquet files
GNU General Public License v3.0
687 stars 82 forks source link

[BUG] Cannot Open Parquet #96

Closed MCRE-BE closed 7 months ago

MCRE-BE commented 8 months ago

Parquet Viewer Version What version of Parquet Viewer are you experiencing the issue with? 2.8.0.3.

Where was the parquet file created? pyarrow

Sample File Upload a sample file so the issue can be debugged!

Describe the bug See below. I guess it has to do with a change in how the metadata is encoded by pyarrow or pandas as I didn't change my code and I could open the files previously.

Something went wrong while processing the schema:

System.Exception: Field schema path not found: `schema/P`
   at ParquetViewer.Engine.ParquetSchemaElement.<GetChild>g__GetChildImpl|15_0(String name)
   at ParquetViewer.Engine.ParquetSchemaElement.GetChild(String name)
   at ParquetViewer.Engine.ParquetEngine.BuildParquetSchemaTree()
   at ParquetViewer.Engine.ParquetEngine.get_ParquetSchemaTree()
   at ParquetViewer.Helpers.ParquetMetadataAnalyzers.ThriftMetadataToJSON(ParquetEngine parquetEngine, Int64 recordCount, Int32 fieldCount)
Arrow Metadata { "Fields": { "Filename": { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Filename", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Date": { "DataType": { "TypeId": 18, "Name": "timestamp", "IsFixedWidth": true }, "Name": "Date", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Site": { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Site", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Article ID": { "DataType": { "TypeId": 8, "Name": "uint64", "IsFixedWidth": true }, "Name": "Article ID", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Description": { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Description", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Supplier": { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Supplier", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "CRP": { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "CRP", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Cat": { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Cat", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Status": { "DataType": { "TypeId": 2, "Name": "uint8", "IsFixedWidth": true }, "Name": "Status", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Storage class": { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Storage class", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "ItemSet": { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "ItemSet", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "CodeWeight": { "DataType": { "TypeId": 2, "Name": "uint8", "IsFixedWidth": true }, "Name": "CodeWeight", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Next week": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Next week", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Total Pre-Ord QTY": { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Total Pre-Ord QTY", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Picked Pre-Ord QTY": { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Picked Pre-Ord QTY", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Total Need J-1": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Reg Need J-1": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Reg Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Promo Need J-1": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Promo Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "P/E/E Need J-1": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "P/E/E Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Needs last 6 days": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Needs last 6 days", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Missings J-1": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Missings J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Missings Promo J-1": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Missings Promo J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "DIO (J-1)": { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "DIO (J-1)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Avail.UP": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Avail.UP", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Total Stock": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "SLOC Intern": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "SLOC Intern", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "SLOC Extern": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "SLOC Extern", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Blocked Stock": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Blocked Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Promo Stock": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Promo Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Total Stock (Pal)": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Total Stock External (Pal)": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock External (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Total Stock Internal (Pal)": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock Internal (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "Stock Value": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Stock Value", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "PA": { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PA", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "PG (AD)": { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PG (AD)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "PV (SM)": { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PV (SM)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "# CV Received J-1": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "# CV Received J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "# Pal recevied J-1": { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "# Pal recevied J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "EA/UP": { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "EA/UP", "IsNullable": true, "HasMetadata": false, "Metadata": null }, "CV/PAL": { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "CV/PAL", "IsNullable": true, "HasMetadata": false, "Metadata": null } }, "FieldsList": [ { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Filename", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 18, "Name": "timestamp", "IsFixedWidth": true }, "Name": "Date", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Site", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 8, "Name": "uint64", "IsFixedWidth": true }, "Name": "Article ID", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Description", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Supplier", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "CRP", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Cat", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 2, "Name": "uint8", "IsFixedWidth": true }, "Name": "Status", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Storage class", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "ItemSet", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 2, "Name": "uint8", "IsFixedWidth": true }, "Name": "CodeWeight", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Next week", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Total Pre-Ord QTY", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Picked Pre-Ord QTY", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Reg Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Promo Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "P/E/E Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Needs last 6 days", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Missings J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Missings Promo J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "DIO (J-1)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Avail.UP", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "SLOC Intern", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "SLOC Extern", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Blocked Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Promo Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock External (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock Internal (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Stock Value", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PA", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PG (AD)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PV (SM)", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "# CV Received J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "# Pal recevied J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "EA/UP", "IsNullable": true, "HasMetadata": false, "Metadata": null }, { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "CV/PAL", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], "FieldsLookup": [ [ { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Filename", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 18, "Name": "timestamp", "IsFixedWidth": true }, "Name": "Date", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Site", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 8, "Name": "uint64", "IsFixedWidth": true }, "Name": "Article ID", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Description", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Supplier", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "CRP", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Cat", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 2, "Name": "uint8", "IsFixedWidth": true }, "Name": "Status", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 13, "Name": "utf8", "IsFixedWidth": false }, "Name": "Storage class", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "ItemSet", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 2, "Name": "uint8", "IsFixedWidth": true }, "Name": "CodeWeight", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Next week", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Total Pre-Ord QTY", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "Picked Pre-Ord QTY", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Reg Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Promo Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "P/E/E Need J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Needs last 6 days", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Missings J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Missings Promo J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "DIO (J-1)", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Avail.UP", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "SLOC Intern", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "SLOC Extern", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Blocked Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Promo Stock", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock External (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Total Stock Internal (Pal)", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "Stock Value", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PA", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PG (AD)", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "PV (SM)", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "# CV Received J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 6, "Name": "uint32", "IsFixedWidth": true }, "Name": "# Pal recevied J-1", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 4, "Name": "uint16", "IsFixedWidth": true }, "Name": "EA/UP", "IsNullable": true, "HasMetadata": false, "Metadata": null } ], [ { "DataType": { "TypeId": 11, "Name": "float", "IsFixedWidth": true }, "Name": "CV/PAL", "IsNullable": true, "HasMetadata": false, "Metadata": null } ] ], "Metadata": { "pandas": "{\u0022index_columns\u0022: [], \u0022column_indexes\u0022: [], \u0022columns\u0022: [{\u0022name\u0022: \u0022Filename\u0022, \u0022field_name\u0022: \u0022Filename\u0022, \u0022pandas_type\u0022: \u0022unicode\u0022, \u0022numpy_type\u0022: \u0022string\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Date\u0022, \u0022field_name\u0022: \u0022Date\u0022, \u0022pandas_type\u0022: \u0022datetime\u0022, \u0022numpy_type\u0022: \u0022datetime64[ns]\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Site\u0022, \u0022field_name\u0022: \u0022Site\u0022, \u0022pandas_type\u0022: \u0022unicode\u0022, \u0022numpy_type\u0022: \u0022string\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Article ID\u0022, \u0022field_name\u0022: \u0022Article ID\u0022, \u0022pandas_type\u0022: \u0022uint64\u0022, \u0022numpy_type\u0022: \u0022uint64\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Description\u0022, \u0022field_name\u0022: \u0022Description\u0022, \u0022pandas_type\u0022: \u0022unicode\u0022, \u0022numpy_type\u0022: \u0022string\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Supplier\u0022, \u0022field_name\u0022: \u0022Supplier\u0022, \u0022pandas_type\u0022: \u0022unicode\u0022, \u0022numpy_type\u0022: \u0022string\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022CRP\u0022, \u0022field_name\u0022: \u0022CRP\u0022, \u0022pandas_type\u0022: \u0022unicode\u0022, \u0022numpy_type\u0022: \u0022string\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Cat\u0022, \u0022field_name\u0022: \u0022Cat\u0022, \u0022pandas_type\u0022: \u0022uint16\u0022, \u0022numpy_type\u0022: \u0022uint16\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Status\u0022, \u0022field_name\u0022: \u0022Status\u0022, \u0022pandas_type\u0022: \u0022uint8\u0022, \u0022numpy_type\u0022: \u0022uint8\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Storage class\u0022, \u0022field_name\u0022: \u0022Storage class\u0022, \u0022pandas_type\u0022: \u0022unicode\u0022, \u0022numpy_type\u0022: \u0022object\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022ItemSet\u0022, \u0022field_name\u0022: \u0022ItemSet\u0022, \u0022pandas_type\u0022: \u0022uint16\u0022, \u0022numpy_type\u0022: \u0022uint16\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022CodeWeight\u0022, \u0022field_name\u0022: \u0022CodeWeight\u0022, \u0022pandas_type\u0022: \u0022uint8\u0022, \u0022numpy_type\u0022: \u0022uint8\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Next week\u0022, \u0022field_name\u0022: \u0022Next week\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Total Pre-Ord QTY\u0022, \u0022field_name\u0022: \u0022Total Pre-Ord QTY\u0022, \u0022pandas_type\u0022: \u0022uint16\u0022, \u0022numpy_type\u0022: \u0022uint16\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Picked Pre-Ord QTY\u0022, \u0022field_name\u0022: \u0022Picked Pre-Ord QTY\u0022, \u0022pandas_type\u0022: \u0022uint16\u0022, \u0022numpy_type\u0022: \u0022uint16\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Total Need J-1\u0022, \u0022field_name\u0022: \u0022Total Need J-1\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Reg Need J-1\u0022, \u0022field_name\u0022: \u0022Reg Need J-1\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Promo Need J-1\u0022, \u0022field_name\u0022: \u0022Promo Need J-1\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022P/E/E Need J-1\u0022, \u0022field_name\u0022: \u0022P/E/E Need J-1\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Needs last 6 days\u0022, \u0022field_name\u0022: \u0022Needs last 6 days\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Missings J-1\u0022, \u0022field_name\u0022: \u0022Missings J-1\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Missings Promo J-1\u0022, \u0022field_name\u0022: \u0022Missings Promo J-1\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022DIO (J-1)\u0022, \u0022field_name\u0022: \u0022DIO (J-1)\u0022, \u0022pandas_type\u0022: \u0022uint16\u0022, \u0022numpy_type\u0022: \u0022uint16\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Avail.UP\u0022, \u0022field_name\u0022: \u0022Avail.UP\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Total Stock\u0022, \u0022field_name\u0022: \u0022Total Stock\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022SLOC Intern\u0022, \u0022field_name\u0022: \u0022SLOC Intern\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022SLOC Extern\u0022, \u0022field_name\u0022: \u0022SLOC Extern\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Blocked Stock\u0022, \u0022field_name\u0022: \u0022Blocked Stock\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Promo Stock\u0022, \u0022field_name\u0022: \u0022Promo Stock\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Total Stock (Pal)\u0022, \u0022field_name\u0022: \u0022Total Stock (Pal)\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Total Stock External (Pal)\u0022, \u0022field_name\u0022: \u0022Total Stock External (Pal)\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Total Stock Internal (Pal)\u0022, \u0022field_name\u0022: \u0022Total Stock Internal (Pal)\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022Stock Value\u0022, \u0022field_name\u0022: \u0022Stock Value\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022PA\u0022, \u0022field_name\u0022: \u0022PA\u0022, \u0022pandas_type\u0022: \u0022float32\u0022, \u0022numpy_type\u0022: \u0022float32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022PG (AD)\u0022, \u0022field_name\u0022: \u0022PG (AD)\u0022, \u0022pandas_type\u0022: \u0022float32\u0022, \u0022numpy_type\u0022: \u0022float32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022PV (SM)\u0022, \u0022field_name\u0022: \u0022PV (SM)\u0022, \u0022pandas_type\u0022: \u0022float32\u0022, \u0022numpy_type\u0022: \u0022float32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022# CV Received J-1\u0022, \u0022field_name\u0022: \u0022# CV Received J-1\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022# Pal recevied J-1\u0022, \u0022field_name\u0022: \u0022# Pal recevied J-1\u0022, \u0022pandas_type\u0022: \u0022uint32\u0022, \u0022numpy_type\u0022: \u0022uint32\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022EA/UP\u0022, \u0022field_name\u0022: \u0022EA/UP\u0022, \u0022pandas_type\u0022: \u0022uint16\u0022, \u0022numpy_type\u0022: \u0022uint16\u0022, \u0022metadata\u0022: null}, {\u0022name\u0022: \u0022CV/PAL\u0022, \u0022field_name\u0022: \u0022CV/PAL\u0022, \u0022pandas_type\u0022: \u0022float32\u0022, \u0022numpy_type\u0022: \u0022float32\u0022, \u0022metadata\u0022: null}], \u0022creator\u0022: {\u0022library\u0022: \u0022pyarrow\u0022, \u0022version\u0022: \u002212.0.1\u0022}, \u0022pandas_version\u0022: \u00222.0.3\u0022}" }, "HasMetadata": true }

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

MCRE-BE commented 7 months ago

@mukunku : I guess you are busy? Any idea when you might have time to look at the issue?

mukunku commented 7 months ago

@MCRE-BE Been quite busy lately. Will try take a look this week.

MCRE-BE commented 7 months ago

Lovely !

MCRE-BE commented 7 months ago

Confirmed, bug is still present in 2.8.1.0 image

mukunku commented 7 months ago

Do you have a sample file I can look at?

mukunku commented 7 months ago

@MCRE-BE Not sure if I can solve this without a sample file. Also please try v2.8.1.2 out to see if that version helps.

mukunku commented 7 months ago

I just realized this is the same issue as #94

image

So your issue should be fixed with v2.8.1.2

MCRE-BE commented 7 months ago

OK. I'll check. I thought it was something with a new release of pyarrow + pandas that maybe wrote masterdata in a strange way. I know I have troubles in other locations/packages.

Thanks !