Support brotli compressed Parquet file

benwatson528 / intellij-avro-parquet-plugin

A Tool Window plugin for IntelliJ that displays Avro and Parquet files and their schemas in JSON.

Apache License 2.0

44 stars 8 forks source link

Support brotli compressed Parquet file #106

Closed hongbo-miao closed 1 year ago

hongbo-miao commented 1 year ago

I saw this ticket https://github.com/benwatson528/intellij-avro-parquet-plugin/issues/96. In my case, I am on macOS.

Currently it shows nothing for brotli compressed Parquet file.

Attached are example Parquet files generated by API like dataframe.to_parquet("myfile-brotli.parquet", engine="pyarrow", compression="brotli"):

myfile-gzip.parquet.zip (Can view successfully)
myfile-brotli.parquet.zip (Cannot view)

It would be great to support, thanks! 😃

benwatson528 commented 1 year ago

Hello, there's been no movement on this from elsewhere so there's nothing I can do here.

hongbo-miao commented 1 year ago

I assume your "no movement on this from elsewhere" means https://github.com/benwatson528/intellij-avro-parquet-plugin/issues/96#issuecomment-1073281471

The plugin contains the Brotli compression class now, but it needs to find the Brotli native library on the host machine to be able to process the file. There are a few libraries out there but none have been updated in the last few years.

Thanks for the update!