search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.21k stars 435 forks source link

[VL] shuffle writter spill not triggered #6863

Closed FelixYBW closed 1 month ago

FelixYBW commented 3 months ago

Backend

VL (Velox)

Bug description

should be sort based shuffle. @marin-ma 

4/08/14 09:47:34 ERROR [Executor task launch worker for task 782.0 in stage 26.0 (TID 102118)] listener.ManagedReservationListener: Error reserving memory from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 8.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled). 
Current config settings: 
    spark.gluten.memory.offHeap.size.in.bytes=50.0 GiB
    spark.gluten.memory.task.offHeap.size.in.bytes=12.5 GiB
    spark.gluten.memory.conservative.task.offHeap.size.in.bytes=6.3 GiB
    spark.memory.offHeap.enabled=true
    spark.gluten.memory.dynamic.offHeap.sizing.enabled=false
Memory consumer stats: 
    Task.102118:                                                Current used bytes: 13.5 GiB, peak bytes:        N/A
    \- Gluten.Tree.158:                                         Current used bytes: 13.5 GiB, peak bytes:   17.7 GiB
       \- root.158:                                             Current used bytes: 13.5 GiB, peak bytes:   17.7 GiB
          +- ShuffleWriter.158:                                 Current used bytes: 13.5 GiB, peak bytes:   13.5 GiB
          |  \- single:                                         Current used bytes: 13.5 GiB, peak bytes:   13.5 GiB
          |     +- root:                                        Current used bytes: 13.5 GiB, peak bytes:   13.5 GiB
          |     |  \- default_leaf:                             Current used bytes: 13.5 GiB, peak bytes:   13.5 GiB
          |     \- gluten::MemoryAllocator:                     Current used bytes:    0.0 B, peak bytes:      0.0 B
          +- WholeStageIterator.158:                            Current used bytes: 22.0 MiB, peak bytes:  152.0 MiB
          |  \- single:                                         Current used bytes: 22.0 MiB, peak bytes:  152.0 MiB
          |     +- root:                                        Current used bytes: 16.2 MiB, peak bytes:  152.0 MiB
          |     |  +- task.Gluten_Stage_26_TID_102118_VTID_216: Current used bytes: 16.2 MiB, peak bytes:  152.0 MiB
          |     |  |  +- node.2:                                Current used bytes: 16.1 MiB, peak bytes:  152.0 MiB
          |     |  |  |  \- op.2.1.0.Aggregation:               Current used bytes: 16.1 MiB, peak bytes:  128.6 MiB
          |     |  |  +- node.5:                                Current used bytes: 78.0 KiB, peak bytes:   28.0 MiB
          |     |  |  |  +- op.5.0.0.HashProbe:                 Current used bytes: 78.0 KiB, peak bytes:   94.0 KiB
          |     |  |  |  \- op.5.1.0.HashBuild:                 Current used bytes:    0.0 B, peak bytes:  132.0 KiB
          |     |  |  +- node.4:                                Current used bytes:  6.3 KiB, peak bytes: 1024.0 KiB
          |     |  |  |  \- op.4.1.0.FilterProject:             Current used bytes:  6.3 KiB, peak bytes:    6.5 KiB
          |     |  |  +- node.7:                                Current used bytes:    0.0 B, peak bytes: 1024.0 KiB
          |     |  |  |  \- op.7.0.0.FilterProject:             Current used bytes:    0.0 B, peak bytes:   12.0 KiB
          |     |  |  +- node.1:                                Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     |  |  |  \- op.1.1.0.ValueStream:               Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     |  |  +- node.6:                                Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     |  |  |  \- op.6.0.0.FilterProject:             Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     |  |  \- node.0:                                Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     |  |     \- op.0.0.0.ValueStream:               Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     |  \- default_leaf:                             Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     \- gluten::MemoryAllocator:                     Current used bytes:    0.0 B, peak bytes:      0.0 B
          +- ArrowContextInstance.203:                          Current used bytes:  8.0 MiB, peak bytes:    8.0 MiB
          +- IndicatorVectorBase#init.158:                      Current used bytes:    0.0 B, peak bytes:      0.0 B
          |  \- single:                                         Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     +- root:                                        Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     |  \- default_leaf:                             Current used bytes:    0.0 B, peak bytes:      0.0 B
          |     \- gluten::MemoryAllocator:                     Current used bytes:    0.0 B, peak bytes:      0.0 B
          +- ShuffleReader.80.OverAcquire.0:                    Current used bytes:    0.0 B, peak bytes:    4.8 MiB
          +- WholeStageIterator.158.OverAcquire.0:              Current used bytes:    0.0 B, peak bytes:   45.6 MiB
          +- ArrowContextInstance.204:                          Current used bytes:    0.0 B, peak bytes:      0.0 B
          +- ShuffleReader.80:                                  Current used bytes:    0.0 B, peak bytes:   16.0 MiB
          |  \- single:                                         Current used bytes:    0.0 B, peak bytes:   16.0 MiB
          |     +- gluten::MemoryAllocator:                     Current used bytes:    0.0 B, peak bytes: 1648.9 KiB
          |     \- root:                                        Current used bytes:    0.0 B, peak bytes: 1024.0 KiB
          |        \- default_leaf:                             Current used bytes:    0.0 B, peak bytes:  240.0 KiB
          +- ShuffleWriter.158.OverAcquire.0:                   Current used bytes:    0.0 B, peak bytes:    4.1 GiB
          \- IndicatorVectorBase#init.158.OverAcquire.0:        Current used bytes:    0.0 B, peak bytes:      0.0 B

    at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
    at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:49)
    at org.apache.gluten.vectorized.ShuffleWriterJniWrapper.nativeEvict(Native Method)
    at org.apache.spark.shuffle.ColumnarShuffleWriter$$anon$1.spill(ColumnarShuffleWriter.scala:173)
    at org.apache.gluten.memory.memtarget.Spillers$AppendableSpillerList.spill(Spillers.java:86)
    at org.apache.gluten.memory.memtarget.Spillers$WithMinSpillSize.spill(Spillers.java:66)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:80)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:55)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:73)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:55)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:73)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets.spillTree(TreeMemoryTargets.java:55)
    at org.apache.gluten.memory.memtarget.spark.TreeMemoryConsumer.spill(TreeMemoryConsumer.java:115)
    at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:213)
    at org.apache.spark.memory.MemoryConsumer.acquireMemory(MemoryConsumer.java:136)
    at org.apache.gluten.memory.memtarget.spark.TreeMemoryConsumer.borrow(TreeMemoryConsumer.java:65)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets$Node.borrow0(TreeMemoryTargets.java:129)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets$Node.borrow(TreeMemoryTargets.java:121)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets$Node.borrow0(TreeMemoryTargets.java:129)
    at org.apache.gluten.memory.memtarget.TreeMemoryTargets$Node.borrow(TreeMemoryTargets.java:121)
    at org.apache.gluten.memory.memtarget.OverAcquire.borrow(OverAcquire.java:59)
    at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:35)
    at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:49)
    at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeNext(Native Method)
    at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nextInternal(ColumnarBatchOutIterator.java:66)
    at org.apache.gluten.vectorized.GeneralOutIterator.next(GeneralOutIterator.java:46)
    at org.apache.gluten.vectorized.ColumnarBatchSerializerInstance$TaskDeserializationStream.liftedTree1$1(ColumnarBatchSerializer.scala:177)
    at org.apache.gluten.vectorized.ColumnarBatchSerializerInstance$TaskDeserializationStream.readValue(ColumnarBatchSerializer.scala:176)
    at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:188)
    at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:185)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:488)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:31)
    at org.apache.gluten.vectorized.GeneralInIterator.hasNext(GeneralInIterator.java:31)
    at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
    at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:61)
    at org.apache.gluten.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
    at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
    at org.apache.gluten.utils.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
    at org.apache.gluten.utils.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
    at org.apache.gluten.utils.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
    at org.apache.gluten.utils.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:136)
    at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:239)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1471)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

marin-ma commented 3 months ago

@FelixYBW Could you provide the full stack trace of the exception?

FelixYBW commented 3 months ago

Done

marin-ma commented 3 months ago

Shuffle spill is triggered but it fails to allocate extra memory during spill

    at org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
    at org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:49)
    at org.apache.gluten.vectorized.ShuffleWriterJniWrapper.nativeEvict(Native Method)
    at org.apache.spark.shuffle.ColumnarShuffleWriter$$anon$1.spill(ColumnarShuffleWriter.scala:173)

The memory allocation likely stems from compression. It can be fixed via pre-allocating the compression buffer.

FelixYBW commented 1 month ago

fixed