search

mukunku / ParquetViewer

Simple Windows desktop application for viewing & querying Apache Parquet files
GNU General Public License v3.0
687 stars 82 forks source link

[BUG] Opening ParquetViewer to an empty view #59

Closed MCRE-BE closed 1 year ago

MCRE-BE commented 1 year ago

Parquet Viewer Version What version of Parquet Viewer are you experiencing the issue with? 2.3.6.22567

Where was the parquet file created? Apache Spark, Hive, Java, C#, pyarrow, etc. Python through pandas with pyarrow backend

Sample File Upload a sample file so the issue can be debugged! DLIx12_test.zip

Describe the bug A clear and concise description of what the bug is. Opening file in ParquetViewer gives this as output. No idea what is wrong. Can it be linked to pyarrow version 9? I have no problem opening same sized file for version 8.

image

Screenshots If applicable, add screenshots to help explain your problem.

Thrift Metadata > { > "Version": 2, > "Num_rows": 3835405, > "Created_by": "parquet-cpp-arrow version 9.0.0", > "Schema": [ > { > "Field_id": 0, > "Name": "Transmission_Number", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Line_Number", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "DC", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "GOLD_Article", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "GOLD_Storage_Loc", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "GOLD_Logistic_Variant", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "SAP_Article", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Delivery_Quantity_Base_Unit", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Base_UOM", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Delivery_Quantity_Preparation_Unit", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Preparation_UOM", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Client", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Shipping_Date", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "GOLD_Shipping_Id", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Line_Id", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "SAP_OBD_Number", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Article_Managed_by_Unit", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Shipped_Net_Weight", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Missing_Quantity", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Missing_Weight", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Pallet_Number", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Mother_Pallet", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Daughter_Pallet", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Code_Picker", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Code_Loader", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Route", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Tour_Rank", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Promo_Number", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Full_Pallet", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Depot_Origine_Transfer", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Depot_Preparation", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Order_Date", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Empty_Included", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Empty_Article_Number_Pallet", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Empty_Article_Number_CV", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Crate_Number", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Warehouse_number", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Child_OBD_Number", > "Type": "INT64", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Missing_Motivation", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Promo_Week", > "Type": "DOUBLE", > "Type_length": 0, > "LogicalType": null, > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Delivery_Type", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > }, > { > "Field_id": 0, > "Name": "Source_Name", > "Type": "BYTE_ARRAY", > "Type_length": 0, > "LogicalType": "LogicalType(STRING: StringType())", > "Scale": 0, > "Precision": 0, > "Repetition_type": "OPTIONAL", > "Converted_type": "UTF8" > } > ] > }
Arrow: Schema > { > "Fields": { > "Transmission_Number": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "Transmission_Number", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Line_Number": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "Line_Number", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "DC": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "DC", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "GOLD_Article": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "GOLD_Article", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "GOLD_Storage_Loc": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "GOLD_Storage_Loc", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "GOLD_Logistic_Variant": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "GOLD_Logistic_Variant", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "SAP_Article": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "SAP_Article", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Delivery_Quantity_Base_Unit": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "Delivery_Quantity_Base_Unit", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Base_UOM": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Base_UOM", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Delivery_Quantity_Preparation_Unit": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "Delivery_Quantity_Preparation_Unit", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Preparation_UOM": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Preparation_UOM", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Client": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Client", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Shipping_Date": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Shipping_Date", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "GOLD_Shipping_Id": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "GOLD_Shipping_Id", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Line_Id": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Line_Id", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "SAP_OBD_Number": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "SAP_OBD_Number", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Article_Managed_by_Unit": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Article_Managed_by_Unit", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Shipped_Net_Weight": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Shipped_Net_Weight", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Missing_Quantity": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "Missing_Quantity", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Missing_Weight": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Missing_Weight", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Pallet_Number": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Pallet_Number", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Mother_Pallet": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Mother_Pallet", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Daughter_Pallet": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Daughter_Pallet", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Code_Picker": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Code_Picker", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Code_Loader": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Code_Loader", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Route": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Route", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Tour_Rank": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Tour_Rank", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Promo_Number": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Promo_Number", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Full_Pallet": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Full_Pallet", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Depot_Origine_Transfer": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Depot_Origine_Transfer", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Depot_Preparation": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Depot_Preparation", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Order_Date": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Order_Date", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Empty_Included": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Empty_Included", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Empty_Article_Number_Pallet": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Empty_Article_Number_Pallet", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Empty_Article_Number_CV": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Empty_Article_Number_CV", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Crate_Number": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Crate_Number", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Warehouse_number": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Warehouse_number", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Child_OBD_Number": { > "DataType": { > "TypeId": 9, > "Name": "int64", > "BitWidth": 64, > "IsSigned": true, > "IsFixedWidth": true > }, > "Name": "Child_OBD_Number", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Missing_Motivation": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Missing_Motivation", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Promo_Week": { > "DataType": { > "TypeId": 12, > "Name": "double", > "BitWidth": 64, > "IsSigned": true, > "Precision": 2, > "IsFixedWidth": true > }, > "Name": "Promo_Week", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Delivery_Type": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Delivery_Type", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > }, > "Source_Name": { > "DataType": { > "TypeId": 13, > "Name": "utf8", > "IsFixedWidth": false > }, > "Name": "Source_Name", > "IsNullable": true, > "HasMetadata": false, > "Metadata": null > } > }, > "Metadata": { > "pandas": "{\"index_columns\": [{\"kind\": \"range\", \"name\": null, \"start\": 0, \"stop\": 3835405, \"step\": 1}], \"column_indexes\": [{\"name\": null, \"field_name\": null, \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": {\"encoding\": \"UTF-8\"}}], \"columns\": [{\"name\": \"Transmission_Number\", \"field_name\": \"Transmission_Number\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"Line_Number\", \"field_name\": \"Line_Number\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"DC\", \"field_name\": \"DC\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"GOLD_Article\", \"field_name\": \"GOLD_Article\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"GOLD_Storage_Loc\", \"field_name\": \"GOLD_Storage_Loc\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"GOLD_Logistic_Variant\", \"field_name\": \"GOLD_Logistic_Variant\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"SAP_Article\", \"field_name\": \"SAP_Article\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"Delivery_Quantity_Base_Unit\", \"field_name\": \"Delivery_Quantity_Base_Unit\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"Base_UOM\", \"field_name\": \"Base_UOM\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Delivery_Quantity_Preparation_Unit\", \"field_name\": \"Delivery_Quantity_Preparation_Unit\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"Preparation_UOM\", \"field_name\": \"Preparation_UOM\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Client\", \"field_name\": \"Client\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Shipping_Date\", \"field_name\": \"Shipping_Date\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"GOLD_Shipping_Id\", \"field_name\": \"GOLD_Shipping_Id\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"Line_Id\", \"field_name\": \"Line_Id\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"SAP_OBD_Number\", \"field_name\": \"SAP_OBD_Number\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"Article_Managed_by_Unit\", \"field_name\": \"Article_Managed_by_Unit\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Shipped_Net_Weight\", \"field_name\": \"Shipped_Net_Weight\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Missing_Quantity\", \"field_name\": \"Missing_Quantity\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"Missing_Weight\", \"field_name\": \"Missing_Weight\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Pallet_Number\", \"field_name\": \"Pallet_Number\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Mother_Pallet\", \"field_name\": \"Mother_Pallet\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Daughter_Pallet\", \"field_name\": \"Daughter_Pallet\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Code_Picker\", \"field_name\": \"Code_Picker\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Code_Loader\", \"field_name\": \"Code_Loader\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Route\", \"field_name\": \"Route\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Tour_Rank\", \"field_name\": \"Tour_Rank\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Promo_Number\", \"field_name\": \"Promo_Number\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Full_Pallet\", \"field_name\": \"Full_Pallet\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Depot_Origine_Transfer\", \"field_name\": \"Depot_Origine_Transfer\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Depot_Preparation\", \"field_name\": \"Depot_Preparation\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Order_Date\", \"field_name\": \"Order_Date\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Empty_Included\", \"field_name\": \"Empty_Included\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Empty_Article_Number_Pallet\", \"field_name\": \"Empty_Article_Number_Pallet\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Empty_Article_Number_CV\", \"field_name\": \"Empty_Article_Number_CV\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Crate_Number\", \"field_name\": \"Crate_Number\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Warehouse_number\", \"field_name\": \"Warehouse_number\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Child_OBD_Number\", \"field_name\": \"Child_OBD_Number\", \"pandas_type\": \"int64\", \"numpy_type\": \"int64\", \"metadata\": null}, {\"name\": \"Missing_Motivation\", \"field_name\": \"Missing_Motivation\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Promo_Week\", \"field_name\": \"Promo_Week\", \"pandas_type\": \"float64\", \"numpy_type\": \"float64\", \"metadata\": null}, {\"name\": \"Delivery_Type\", \"field_name\": \"Delivery_Type\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}, {\"name\": \"Source_Name\", \"field_name\": \"Source_Name\", \"pandas_type\": \"unicode\", \"numpy_type\": \"object\", \"metadata\": null}], \"creator\": {\"library\": \"pyarrow\", \"version\": \"9.0.0\"}, \"pandas_version\": \"1.4.3\"}" > }, > "HasMetadata": true > }

Additional context Add any other context about the problem here.

Note: This tool relies on the parquet-dotnet library for all the actual Parquet processing. So any issues where that library cannot process a parquet file will not be addressed by us. Please open a ticket on that library's repo to address such issues.

MCRE-BE commented 1 year ago

Can it be linked to? this comment by @eedokl in https://github.com/mukunku/ParquetViewer/issues/37#issuecomment-1209167417_

MCRE-BE commented 1 year ago

BTW. I saw the 2.3.7 pre-release, same bug image

MCRE-BE commented 1 year ago

@mukunku is it an easy fix or is it a big issue? Love your app, but more and more unusable as I update my files. Shamefully I cannot assist in a fix :-(

mukunku commented 1 year ago

@mukunku is it an easy fix or is it a big issue? Love your app, but more and more unusable as I update my files. Shamefully I cannot assist in a fix :-(

Unfortunately there's no easy fix for this. Parquet.net doesn't support v2.4 and I can't find any other good alternatives. I just checked the repo and there was a release literally 10 hours ago after more than a year but no mention of v2.4 support. But maybe there's still hope?

I realize without adding support for these files this tool's relevance will fade so I'm trying my best but not sure how much I can accomplish.

MCRE-BE commented 1 year ago

Shame :-( I'll monitor the repo then and I'll see when you are able to push an update. Likely someone on their end will have figured that one out as well and will do the needed changes. I'll stick with Python then for the moment.

Thanks for all the efforts !

mukunku commented 1 year ago

@MCRE-BE Looks like Christmas came early! parquet-dotnet released v4.0.0 and I created an experimental release: https://github.com/mukunku/ParquetViewer/releases/tag/v2.4.1

Looks like it's able to open your sample file now. Let me know how it goes with other files!

MCRE-BE commented 1 year ago

Hey indeed ! It seems to be working now. Loading Data is much much slower (to never stopping), but I guess that's because of the major refactor and this being experimental.

Top ! I'll close the bug and report in PR #60 if I see any things that need sharing.

mukunku commented 1 year ago

Hey indeed ! It seems to be working now. Loading Data is much much slower (to never stopping), but I guess that's because of the major refactor and this being experimental.

Top ! I'll close the bug and report in PR #60 if I see any things that need sharing.

There was a bug with file loading. Try the latest exe out: https://github.com/mukunku/ParquetViewer/releases/tag/v2.4.1

MCRE-BE commented 1 year ago

The new release fixed that bug. Thanks 😃