search

mukunku / ParquetViewer

Simple Windows desktop application for viewing & querying Apache Parquet files
GNU General Public License v3.0
689 stars 82 forks source link

Exception when opening file (1,6 MB) #16

Closed staticbool closed 3 years ago

staticbool commented 3 years ago

I am exporting a table from SQL Server to parquet and back. I am using Parquet.NET v 3.0 to perform the export and it seems to run normally. When opening one file (~266KB) with the viewer it just works fine. Another "larger" file (1,6MB) would throw the following exception:

Exception ParquetViewer

after the column selection dialog.

I investigated further and this only happens if I select String columns that had been of type "varchar" on SQL server. Any "nvarchar" column works just fine.

However when passed to the writer they are converted to regular C# strings before, so encoding should not normally be an issue.

mukunku commented 3 years ago

Which version of Parquet Viewer are you using? I'd recommend 2.1: https://github.com/mukunku/ParquetViewer/releases/tag/v2.1

If that doesn't work I'd recommend trying a more recent version of Parquet.Net to generate the file as maybe v3.0 files are not compatible with v3.3.8 that Parquet Viewer 2.1 is using.

I've opened 10gb parquet files before with no issues.

staticbool commented 3 years ago

I am using the most recent version of Parquet Viewer. I thought that would also be the case with the Parquet.NET library, but updating it again did the trick.