search

xerial / snappy-java

Snappy compressor/decompressor for Java
Apache License 2.0
1.04k stars 232 forks source link

Exception writing repeated INT64 parquet column with purejava implementation #282

Closed liorchaga closed 1 year ago

liorchaga commented 3 years ago

I'm trying to read this file saved with spark 2.2, parquet 1.8.2 with snappy-java 1.1.2.6

Reading with spark 3.1.1, parquet 1.10.1 snappy-java 1.1.8.3 with purejava enabled, I'm hitting an exception when saving back to parquet:

Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file file:///sample.snappy.parquet
  at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:251)
  at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
  at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:37)
  at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
  at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
  at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173)
  ... 12 more
Caused by: java.lang.IllegalArgumentException
  at java.nio.Buffer.limit(Buffer.java:275)
  at org.xerial.snappy.Snappy.uncompress(Snappy.java:553)
  at org.apache.parquet.hadoop.codec.SnappyDecompressor.decompress(SnappyDecompressor.java:69)
  at org.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:51)
  at java.io.DataInputStream.readFully(DataInputStream.java:195)
  at java.io.DataInputStream.readFully(DataInputStream.java:169)
  at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:263)
  at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:214)
  at org.apache.parquet.bytes.BytesInput.toInputStream(BytesInput.java:223)
  at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary.<init>(PlainValuesDictionary.java:154)
  at org.apache.parquet.column.Encoding$1.initDictionary(Encoding.java:94)
  at org.apache.parquet.column.Encoding$4.initDictionary(Encoding.java:147)
  at org.apache.parquet.column.impl.ColumnReaderImpl.<init>(ColumnReaderImpl.java:339)
  at org.apache.parquet.column.impl.ColumnReadStoreImpl.newMemColumnReader(ColumnReadStoreImpl.java:80)
  at org.apache.parquet.column.impl.ColumnReadStoreImpl.getColumnReader(ColumnReadStoreImpl.java:75)
  at org.apache.parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:271)
  at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:147)
  at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:109)
  at org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:165)
  at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:109)
  at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:137)
  at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222)
ebuildy commented 2 years ago

You should use a native library!

xerial commented 1 year ago

Removed pure-java support in https://github.com/xerial/snappy-java/releases/tag/v1.1.9.0

Hc747 commented 9 months ago

@xerial, why was pure-java support removed?

xerial commented 9 months ago

@Hc747 It was removed because it was buggy and caused data corruption.

Hc747 commented 9 months ago

@Hc747 It was removed because it was buggy and caused data corruption.

Interesting - it seems like the native library is no longer an option for alpine, so having the pure java implementation as a fallback would've been quite nice.