search

benwatson528 / intellij-avro-parquet-plugin

A Tool Window plugin for IntelliJ that displays Avro and Parquet files and their schemas in JSON.
Apache License 2.0
43 stars 9 forks source link

Parquet file not loaded in Pycharm MacOS #65

Closed shubhamlondhe1996 closed 3 years ago

shubhamlondhe1996 commented 3 years ago

Unable to process Parquet file

org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
    at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195)
    at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167)
    at org.xerial.snappy.Snappy.init(Snappy.java:69)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
    at org.apache.parquet.hadoop.codec.SnappyDecompressor.decompress(SnappyDecompressor.java:62)
    at org.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:51)
    at java.base/java.io.DataInputStream.readFully(DataInputStream.java:200)
    at java.base/java.io.DataInputStream.readFully(DataInputStream.java:170)
    at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:279)
    at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:230)
    at org.apache.parquet.bytes.BytesInput.toInputStream(BytesInput.java:239)
    at org.apache.parquet.column.impl.ColumnReaderBase.readPageV1(ColumnReaderBase.java:650)
    at org.apache.parquet.column.impl.ColumnReaderBase.access$300(ColumnReaderBase.java:57)
    at org.apache.parquet.column.impl.ColumnReaderBase$3.visit(ColumnReaderBase.java:593)
    at org.apache.parquet.column.impl.ColumnReaderBase$3.visit(ColumnReaderBase.java:590)
    at org.apache.parquet.column.page.DataPageV1.accept(DataPageV1.java:120)
    at org.apache.parquet.column.impl.ColumnReaderBase.readPage(ColumnReaderBase.java:590)
    at org.apache.parquet.column.impl.ColumnReaderBase.checkRead(ColumnReaderBase.java:564)
    at org.apache.parquet.column.impl.ColumnReaderBase.consume(ColumnReaderBase.java:705)
    at org.apache.parquet.column.impl.ColumnReaderImpl.consume(ColumnReaderImpl.java:30)
    at org.apache.parquet.column.impl.ColumnReaderImpl.<init>(ColumnReaderImpl.java:47)
    at org.apache.parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:84)
    at org.apache.parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:271)
    at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:147)
    at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:109)
    at org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:165)
    at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:109)
    at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:137)
    at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
    at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:132)
    at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
    at uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getRecords(ParquetFileReader.java:44)
    at uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:180)
    at uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:171)
    at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
benwatson528 commented 3 years ago

Hi,

This plugin is tested with Snappy files on Macs. I'm afraid that this is likely an issue with the JRE setup on your machine - Google brings up similar issues - https://github.com/xerial/snappy-java/issues/38.