search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.22k stars 437 forks source link

[CH] NPE when disable spark.shuffle.compress #7983

Closed exmy closed 2 days ago

exmy commented 3 days ago

Backend

CH (ClickHouse)

Bug description

org.apache.gluten.exception.GlutenException: java.lang.NullPointerException
        at org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager.getWriter(CelebornShuffleManager.java:344)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:57)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
        at org.apache.spark.shuffle.CHCelebornColumnarShuffleWriter.<init>(CHCelebornColumnarShuffleWriter.scala:53)
        at org.apache.spark.shuffle.CHCelebornColumnarShuffleWriterFactory.createShuffleWriterInstance(CHCelebornColumnarShuffleWriterFactory.scala:44)
        at org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager.getWriter(CelebornShuffleManager.java:334)
        ... 10 more

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response