search

benwatson528 / intellij-avro-parquet-plugin

A Tool Window plugin for IntelliJ that displays Avro and Parquet files and their schemas in JSON.
Apache License 2.0
43 stars 9 forks source link

Read parquet file compressed with snappy #20

Closed bioinfornatics closed 4 years ago

bioinfornatics commented 4 years ago

Dear I am using pycharm 2019.3.3

once my project loaded from pycarm if I double click on .parquet or .parquet.snappy the binary file is open from an external text tool (gedit)

How to display parquet file using your plugins ?

Thanks for your plugin

Best regards

benwatson528 commented 4 years ago

Hello, the plugin creates a Tool Window that should appear at the bottom of your IDE (or can be opened via the menu bar otherwise). You can then drag .avro and .parquet files into it.

bioinfornatics commented 4 years ago

@benwatson528 thanks for your quick help :-) Indeed that works for file which end with parquet (I missed this the viewer from the bottom bar)

But not for *.parquet.snappy

On linux it is possible to use magic file to guessing the file type:

 $ file  example.parquet.snappy 
example.parquet.snappy: Apache Parquet

Maybe that could be an enhancement

benwatson528 commented 4 years ago

If you remove the ".snappy" from the end then is it successfully read?

On Thu, 19 Mar 2020, 08:31 jonathan MERCIER, notifications@github.com wrote:

@benwatson528 https://github.com/benwatson528 thanks for your quick help :-) Indeed that works for file which end with parquet but not for *.parquet.snappy

On linux it st possible to use magic file to guessing the fyle type:

$ file example.parquet.snappy example.parquet.snappy: Apache Parquet

Maybe that could be an enhancement

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/benwatson528/intellij-avro-parquet-plugin/issues/20#issuecomment-601051556, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPNI2KD5INA4SCZXEK52XLRIHJ57ANCNFSM4LPCYJDA .

bioinfornatics commented 4 years ago

If you remove the ".snappy" from the end then is it successfully read?

yes that works. The tool guess the file type using file extension and not its magic number.

Anyway to me it is ok. Thanks have a good day

benwatson528 commented 4 years ago

Well I guess I have nothing better to do during quarantine :p. This seems like a good feature to include, thanks for the suggestion. Time to dive into the murky world of Java's magic number handling.

benwatson528 commented 4 years ago

I've uploaded the changes to IntelliJ, it might take a day or two to be accepted. Let me know if it works for you.

benwatson528 commented 4 years ago

@bioinfornatics the changes have approved and the plugin is now live - you should get a prompt to update next time you open Pycharm.