search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.14k stars 415 forks source link

[VL] Velox cache feature broken: MemoryAllocator::allocateNonContiguous crush #4622

Open xingnailu opened 7 months ago

xingnailu commented 7 months ago

Backend

VL (Velox)

Bug description

I am using Glue + Velox, and the data is stored on S3. However, I found that a large portion of the task running time is spent on reading remote data. After discovering the local cache feature, I intend to enable it to verify its effectiveness. However, I encountered the following error:

C [libvelox.so+0x602f34e] facebook::velox::memory::MemoryAllocator::allocateNonContiguous(unsigned long, facebook::velox::memory::Allocation&, std::function<void (long, bool)>, unsigned long)+0x3e I found the track log below, but it doesn't seem to show specific errors, it just runs to this line of code.

Spark version

None

Spark configurations

spark submit config:

$SPARK_HOME/bin/spark-submit \ --master $K8S_MASTER \ --deploy-mode cluster \ --name spark-tpch-gluten-1g-$i \ --conf spark.executor.instances=2 \ --conf spark.executor.memory=10G \ --conf spark.executor.cores=6 \ --conf spark.driver.cores=4 \ --conf spark.driver.memory=4g \ --conf spark.driver.maxResultSize=1g \ --conf spark.eventLog.enabled=true \ --conf spark.eventLog.dir=$EVENTLOG_DIR \ --conf spark.hadoop.fs.s3a.access.key= --conf spark.hadoop.fs.s3a.secret.key= --conf spark.hadoop.fs.s3a.endpoint= --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \ --conf spark.hadoop.fs.s3a.use.instance.credentials=false \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.namespace=$NAMESPACE \ --conf spark.kubernetes.container.image=$IMAGE \ --conf spark.kubernetes.container.image.pullPolicy=Always \ --conf spark.kubernetes.driver.podTemplateFile=$DRIVER_TMPL \ --conf spark.kubernetes.executor.podTemplateFile=$EXEXUTOR_TMPL \ --conf spark.kubernetes.driver.name=spark-tpch-gluten \ --conf spark.sql.files.maxPartitionBytes=128m \ --conf spark.sql.shuffle.partitions=200 \ --conf spark.default.parallelism=200 \ --conf spark.sql.adaptive.enabled=true \ --conf spark.plugins=io.glutenproject.GlutenPlugin \ --conf spark.gluten.loadLibFromJar=true \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=6g \ --conf spark.executor.memoryOverhead=4g \ --conf spark.gluten.sql.debug=true \ --conf spark.gluten.sql.injectNativePlanStringToExplain=true \ --conf spark.gluten.ui.enabled=true \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.gluten.sql.columnar.backend.velox.glogSeverityLevel=0 \ --conf spark.shuffle.service.enabled=false \ --conf spark.sql.adaptive.localShuffleReader.enabled=false \ --conf spark.gluten.sql.columnar.backend.velox.cacheEnabled=true \ --conf spark.gluten.sql.columnar.backend.velox.ssdCacheSize=0 \ --conf spark.gluten.sql.columnar.backend.velox.memCacheSize=1073741824 \

spark UI info: Backend Velox Backend Branch HEAD Backend Revision 8d12a9b36d4f754c3194aa8612f62c0d5395d541 Backend Revision Time 2024-01-11 16:26:31 +0800 GCC Version gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Gluten Branch fix-finalized-s3 Gluten Build Time 2024-01-11T08:45:18Z Gluten Repo URL http://code.oppoer.me/bdc/spark/gluten.git Gluten Revision c03dfeda16e3784e1afc87fab8965c5cf0d618a8 Gluten Revision Time 2024-01-11 16:41:35 +0800 Gluten Version 1.1.0-SNAPSHOT Hadoop Version 2.7.4 Java Version 11.0.21 Scala Version 2.12.15 Spark Version 3.4.1

System information

Velox System Info v0.0.2 Commit: 58d80a1abcaeab5f174e40b4f74fe6b209700cad CMake Version: 3.16.3 System: Arch: C++ Compiler: C++ Compiler Version: C Compiler: C Compiler Version: CMake Prefix Path:

Relevant logs

task log:
2024-02-02T08:13:26.034099154Z stderr F I0202 08:13:26.034029    87 HiveDataSource.cpp:262] No need to check type lowercase modeDATE
2024-02-02T08:13:26.034130125Z stderr F I0202 08:13:26.034093    87 HiveDataSource.cpp:546] Adding split Hive: s3a://xxxx-warehouse/user/hive/warehouse/tpch_1g_orc.db/lineitem/000000_0 4194304 - 4194304
2024-02-02T08:13:26.091046188Z stderr F I0202 08:13:26.090955    87 FileHandle.cpp:50] Generating file handle for: s3a://xxx-warehouse/user/hive/warehouse/tpch_1g_orc.db/lineitem/000000_0 uuid: 1
2024-02-02T08:13:26.09115247Z stdout F #
2024-02-02T08:13:26.091159517Z stdout F # A fatal error has been detected by the Java Runtime Environment:
2024-02-02T08:13:26.09116376Z stdout F #
2024-02-02T08:13:26.091166667Z stdout F #  SIGSEGV (0xb) at pc=0x00007fc66de3934e, pid=16, tid=0x00007fc6931fb700
2024-02-02T08:13:26.091169891Z stdout F #
2024-02-02T08:13:26.091173795Z stdout F # JRE version: OpenJDK Runtime Environment (8.0_382-b05) (build 1.8.0_382-8u382-ga-1~20.04.1-b05)
2024-02-02T08:13:26.091178655Z stdout F # Java VM: OpenJDK 64-Bit Server VM (25.382-b05 mixed mode linux-amd64 compressed oops)
2024-02-02T08:13:26.091182465Z stdout F # Problematic frame:
2024-02-02T08:13:26.094288097Z stdout F # C  [libvelox.so+0x602f34e]  facebook::velox::memory::MemoryAllocator::allocateNonContiguous(unsigned long, facebook::velox::memory::Allocation&, std::function<void (long, bool)>, unsigned long)+0x3e
2024-02-02T08:13:26.094300886Z stdout F #
2024-02-02T08:13:26.094303681Z stdout F # Core dump written. Default location: /opt/spark/work-dir/core or core.16
2024-02-02T08:13:26.094305871Z stdout F #
2024-02-02T08:13:26.094334563Z stdout F # An error report file with more information is saved as:
2024-02-02T08:13:26.094339558Z stdout F # /opt/spark/work-dir/hs_err_pid16.log
2024-02-02T08:13:26.100866576Z stderr F I0202 08:13:26.100792    88 FileSystems.cpp:161] LocalFileSystem::mkdir /var/data/spark-5fcdd701-9856-4651-9609-9f5b0f40f99c/gluten-8bc13c56-c338-472f-aa13-4144218fd4ba/gluten-spill/1e3fb351-95e5-4e9d-a65e-49016b90b277

hs_err_pid.log  print stack trace：
Stack: [0x00007fc6930fb000,0x00007fc6931fc000],  sp=0x00007fc6931f8bf0,  free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libvelox.so+0x602f34e]  facebook::velox::memory::MemoryAllocator::allocateNonContiguous(unsigned long, facebook::velox::memory::Allocation&, std::function<void (long, bool)>, unsigned long)+0x3e

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  io.glutenproject.vectorized.ColumnarBatchOutIterator.nativeHasNext(J)Z+0
j  io.glutenproject.vectorized.ColumnarBatchOutIterator.hasNextInternal()Z+5
j  io.glutenproject.vectorized.GeneralOutIterator.hasNext()Z+1
J 2051 C1 scala.collection.convert.Wrappers$JIteratorWrapper.hasNext()Z (10 bytes) @ 0x00007fc739663db4 [0x00007fc739663c80+0x134]
j  io.glutenproject.utils.InvocationFlowProtection.hasNext()Z+59
j  io.glutenproject.utils.IteratorCompleter.hasNext()Z+4
j  io.glutenproject.utils.PayloadCloser.hasNext()Z+8
j  io.glutenproject.utils.PipelineTimeAccumulator.hasNext()Z+4
j  org.apache.spark.InterruptibleIterator.hasNext()Z+11
j  scala.collection.Iterator$$anon$10.hasNext()Z+4
j  org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(Lscala/collection/Iterator;)V+1
j  org.apache.spark.shuffle.ColumnarShuffleWriter.write(Lscala/collection/Iterator;)V+2
j  org.apache.spark.shuffle.ShuffleWriteProcessor.write(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/ShuffleDependency;JLorg/apache/spark/TaskContext;Lorg/apache/spark/Partition;)Lorg/apache/spark/scheduler/MapStatus;+46
j  org.apache.spark.scheduler.ShuffleMapTask.runTask(Lorg/apache/spark/TaskContext;)Lorg/apache/spark/scheduler/MapStatus;+189
j  org.apache.spark.scheduler.ShuffleMapTask.runTask(Lorg/apache/spark/TaskContext;)Ljava/lang/Object;+2
j  org.apache.spark.TaskContext.runTaskWithListeners(Lorg/apache/spark/scheduler/Task;)Ljava/lang/Object;+2
j  org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;ILscala/collection/immutable/Map;Lscala/Option;)Ljava/lang/Object;+254
j  org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Lorg/apache/spark/executor/Executor$TaskRunner;Lscala/runtime/BooleanRef;)Ljava/lang/Object;+43
j  org.apache.spark.executor.Executor$TaskRunner$$Lambda$746.apply()Ljava/lang/Object;+8
j  org.apache.spark.util.Utils$.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object;+4
j  org.apache.spark.executor.Executor$TaskRunner.run()V+457
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
xingnailu commented 7 months ago

@Yohahaha Do you know what happened?

zhli1142015 commented 7 months ago

I thought this feature may be broken in 1.1.0. Please check our experiments mentioned in https://github.com/oap-project/gluten/pull/4400 and https://github.com/oap-project/gluten/pull/4407. BTW, I think this feature now is not production ready. Hope this help~

zhouyuan commented 7 months ago

@xingnailu thanks for reporting, could you please have a try with the main branch? as @zhli1142015 mentioned, it's not fully tested and should be broken in 1.1 release

thanks, -yuan