search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.31k stars 1.24k forks source link

Buffer overflow exception in `VarByteChunkSVForwardIndexWriter` of Spark data push flow #8638

Open jasperjiaguo opened 2 years ago

jasperjiaguo commented 2 years ago

Our Spark push flow is failing with a table using var length dictionary, but the hadoop job succeeded on the same table. The problem 

03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO - Caused by: java.nio.BufferOverflowException
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at java.nio.ByteBuffer.put(ByteBuffer.java:859)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putBytes(VarByteChunkSVForwardIndexWriter.java:101)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putString(VarByteChunkSVForwardIndexWriter.java:92)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator.putString(SingleValueVarByteRawIndexCreator.java:109)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(SegmentColumnarIndexCreator.java:377)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:244)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at <LinkedIn Spark push job>
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2094)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2094)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.scheduler.Task.run(Task.scala:109)
03-05-2022 21:11:29 PDT validate-pinot-code-spark-avro INFO -   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:429)

The problem can be traced back to at least Commit 21632dadb8cd2d8b77aec523a758d73

jasperjiaguo commented 2 years ago

cc @Jackie-Jiang @siddharthteotia

Jackie-Jiang commented 2 years ago

It think this issue is fixed in #8286. Can you please try the same job with the latest version?

npawar commented 2 years ago

@jasperjiaguo were you able to try with the fix? is this resolved for you?

jasperjiaguo commented 2 years ago

@npawar No. I tested with the latest version having this commit, but still giving me the same error.