Open xingnailu opened 3 months ago
VL (Velox)
I am using Gluten(tag v1.1.1) + Velox + folly + spark3.4.2 + yarn,building with centos8 aarch64, running on aarch64 , while yarn container running with reading s3 data, throw Assertion failure: hp.second == srcChunk->tag(srcI)
None
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS | yarnrm1b-0.yarnrm.bigdata.svc.cluster.local,yarnrm1b-1.yarnrm.bigdata.svc.cluster.local spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES | http://yarnrm1b-0.yarnrm.bigdata.svc.cluster.local:8088/proxy/application_1693383838041_3359264,http://yarnrm1b-1.yarnrm.bigdata.svc.cluster.local:8088/proxy/application_1693383838041_3359264 spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.RM_HA_URLS | yarnrm1b-0.yarnrm.bigdata.svc.cluster.local:8088,yarnrm1b-1.yarnrm.bigdata.svc.cluster.local:8088 spark.plugins | io.glutenproject.GlutenPlugin spark.reducer.maxBlocksInFlightPerAddress | 1000 spark.reducer.maxReqsInFlight | 1000 spark.repl.class.outputDir | /data/data1/yarn/nm/usercache/hive/appcache/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000001/tmp/spark7852604390862141239 spark.repl.class.uri | spark://xxx:44823/classes spark.scheduler.mode | FIFO spark.serializer | org.apache.spark.serializer.KryoSerializer spark.shuffle.consolidateFiles | true spark.shuffle.io.maxRetries | 5 spark.shuffle.io.retryWait | 10 spark.shuffle.manager | org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager spark.shuffle.registration.maxAttempts | 5 spark.shuffle.registration.timeout | 120000 spark.shuffle.service.enabled | false spark.shuffle.useOldFetchProtocol | true spark.speculation | true spark.speculation.interval | 10000 spark.speculation.quantile | 0.95 spark.sql.adaptive.advisoryPartitionSizeInBytes | 134217728 spark.sql.adaptive.enabled | true spark.sql.adaptive.localShuffleReader.enabled | false spark.sql.adaptive.shuffle.targetPostShuffleInputSize | 268435456 spark.sql.autoBroadcastJoinThreshold | 20971520 spark.sql.broadcastTimeout | 1200 spark.sql.catalogImplementation | hive spark.sql.compatible.check.enabled | false spark.sql.extensions | io.glutenproject.GlutenSessionExtensions spark.sql.files.maxPartitionBytes | 268435456 spark.sql.files.openCostInBytes | 8388608 spark.sql.files.readParallelism | 1 spark.sql.hive.caseSensitiveInferenceMode | NEVER_INFER spark.sql.legacy.allowHashOnMapType | true spark.sql.legacy.timeParserPolicy | LEGACY spark.sql.mapKeyDedupPolicy | LAST_WIN spark.sql.orc.compression.codec | zlib spark.sql.parquet.fs.optimized.committer.optimization-enabled | false spark.sql.parquet.writeLegacyFormat | true spark.sql.shuffle.partitions | 400 spark.sql.sources.parallelPartitionDiscovery.parallelism | 10 spark.sql.storeAssignmentPolicy | LEGACY spark.storage.decommission.enabled | true spark.storage.decommission.fallbackStorage.path | s3a://bi.oppo.com/yarn-eks/bigdata-default/ spark.storage.decommission.rddBlocks.enabled | true spark.storage.decommission.shuffleBlocks.enabled | true spark.submit.deployMode | cluster spark.ui.filters | org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter spark.ui.port | 0 spark.yarn.am.extraJavaOptions | -Ddubbo.application.qos.enable=false -Dcom.amazonaws.services.s3.enableV4=true -Djava.net.preferIPv4Stack=true -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:MaxDirectMemorySize=2g -XX:+UseCompressedOops -XX:ParallelGCThreads=8 -XX:ConcGCThreads=4 -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:OnOutOfMemoryError="kill -9 %p" -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Xloggc:/gc.log spark.yarn.am.memory | 2g spark.yarn.am.waitTime | 600s spark.yarn.app.container.log.dir | /data/data1/yarn/container-logs/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000001 spark.yarn.app.id | application_1693383838041_3359264 spark.yarn.archive | s3a:/xxxxapp/spark/spark3.4.2-gluten1.1.1-centos8-arm-20240523-1619.zip spark.yarn.dist.archives | s3a://xxx/app/spark/sparkr/sparkr-20230509.zip#sparkr spark.yarn.dist.files | file:///opt/apache-livy-0.7.1-incubating-SNAPSHOT-bin/conf/yarn/hive-site.xml spark.yarn.historyServer.address | sparkhs:18080 spark.yarn.isPython | true spark.yarn.maxAppAttempts | 1 spark.yarn.priority | 4 spark.yarn.queue | root.wireless_sg.daily spark.yarn.secondary.jars | AdsUDF.2.0.0.jar,appstore_exposure_udf-1.0.jar,bdp_udf-1.0-SNAPSHOT.jar,hive_udf-1.0-jar-ip.jar,hive_udf-1.0-jar-position.jar,hive_udf-1.0.jar,hive_udfs-1.0.0.jar,universe_cdo_expose_obj_opt_format.jar,GeoIpParse.jar,kryo-shaded-4.0.2.jar,livy-api-0.7.0-incubating-SNAPSHOT.jar,livy-rsc-0.7.0-incubating-SNAPSHOT.jar,livy-thriftserver-session-0.7.0-incubating-SNAPSHOT.jar,minlog-1.3.0.jar,netty-all-4.1.47.Final.jar,objenesis-2.5.1.jar,commons-codec-1.9.jar,livy-client-common-0.7.0-incubating-SNAPSHOT.jar,livy-core_2.12-0.7.0-incubating-SNAPSHOT.jar,livy-repl_2.12-0.7.0-incubating-SNAPSHOT.jar,datanucleus-rdbms-4.1.19.jar,datanucleus-core-4.1.17.jar,datanucleus-api-jdo-4.2.4.jar spark.yarn.submit.waitAppCompletion | false
Velox System Info v0.0.2 Commit: 8a935d575aa0a38c3324f7ee98c87b576eb7ad70 CMake Version: 3.20.2 System: Linux-4.18.0-240.10.1.el8_3.aarch64 Arch: aarch64 C++ Compiler: /opt/rh/gcc-toolset-10/root/usr/bin/c++ C++ Compiler Version: 10.3.1 C Compiler: /opt/rh/gcc-toolset-10/root/usr/bin/cc C Compiler Version: 10.3.1 CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt
\nThe results will be copied to your clipboard if xclip is installed.
[2024-05-27 01:45:46.687]Container exited with a non-zero exit code 134. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : /bin/bash: line 1: 2322107 Aborted (core dumped) /usr/lib/jvm/java-1.8.0/bin/java -server -Xmx4096m '-Djava.net.preferIPv6Addresses=false' '-XX:+IgnoreUnrecognizedVMOptions' '--add-opens=java.base/java.lang=ALL-UNNAMED' '--add-opens=java.base/java.lang.invoke=ALL-UNNAMED' '--add-opens=java.base/java.lang.reflect=ALL-UNNAMED' '--add-opens=java.base/java.io=ALL-UNNAMED' '--add-opens=java.base/java.net=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.util=ALL-UNNAMED' '--add-opens=java.base/java.util.concurrent=ALL-UNNAMED' '--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED' '--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED' '--add-opens=java.base/sun.nio.ch=ALL-UNNAMED' '--add-opens=java.base/sun.nio.cs=ALL-UNNAMED' '--add-opens=java.base/sun.security.action=ALL-UNNAMED' '--add-opens=java.base/sun.util.calendar=ALL-UNNAMED' '--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED' '-Djdk.reflect.useDirectMethodHandle=false' '-Ddubbo.application.qos.enable=false' '-Duser.timeZone=GMT+08' '-Dcom.amazonaws.services.s3.enableV4=true' '-Djava.net.preferIPv4Stack=true' '-XX:MetaspaceSize=512m' '-XX:MaxMetaspaceSize=512m' '-XX:MaxDirectMemorySize=1g' '-XX:+UseCompressedOops' '-XX:ParallelGCThreads=8' '-XX:ConcGCThreads=4' '-XX:+UseG1GC' '-XX:SoftRefLRUPolicyMSPerMB=0' '-XX:OnOutOfMemoryError=kill -9 %p' '-verbose:gc' '-XX:+PrintGCDetails' '-XX:+PrintGCTimeStamps' '-XX:+PrintGCDateStamps' '-XX:+PrintHeapAtGC' '-Xloggc:/data/data1/yarn/container-logs/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000003/gc.log' '-XX:MaxDirectMemorySize=3686m' -Djava.io.tmpdir=/data/data1/yarn/nm/usercache/hive/appcache/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000003/tmp '-Dspark.network.timeout=120s' '-Dspark.driver.port=44823' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/data/data1/yarn/container-logs/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000003 org.apache.spark.executor.YarnCoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@100-64-115-176.bigdata.pod.cluster.local:44823 --executor-id 2 --hostname xx.bigdata.pod.cluster.local --cores 4 --app-id application_1693383838041_3359264 --resourceProfileId 0 > /data/data1/yarn/container-logs/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000003/stdout 2> /data/data1/yarn/container-logs/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000003/stderr Last 4096 bytes of stderr : bytes in memory (estimated size 203.7 KiB, free 5.2 GiB) 24/05/27 01:45:35 INFO TorrentBroadcast: Reading broadcast variable 4 took 81 ms 24/05/27 01:45:35 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 612.5 KiB, free 5.2 GiB) 24/05/27 01:45:36 INFO deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 24/05/27 01:45:39 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true. 24/05/27 01:45:39 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type 24/05/27 01:45:39 INFO CheckAllocator: Using DefaultAllocationManager at memory/DefaultAllocationManagerFactory.class 24/05/27 01:45:40 INFO TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 4.0 MiB) 24/05/27 01:45:40 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 33.3 KiB, free 5.2 GiB) 24/05/27 01:45:40 INFO TorrentBroadcast: Reading broadcast variable 1 took 9 ms 24/05/27 01:45:40 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 48.1 KiB, free 5.2 GiB) Assertion failure: hp.second == srcChunk->tag(srcI) Message: File: /usr/local/include/folly/container/detail/F14Table.h Line: 2064 Function: rehashImpl
@PHILO-HE Please take a look
@xingnailu, it may be a bug in folly. Could you re-test it with Gluten main branch? The folly version has been upgraded since 1.1.1.
Backend
VL (Velox)
Bug description
I am using Gluten(tag v1.1.1) + Velox + folly + spark3.4.2 + yarn,building with centos8 aarch64, running on aarch64 , while yarn container running with reading s3 data, throw Assertion failure: hp.second == srcChunk->tag(srcI)
Spark version
None
Spark configurations
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS | yarnrm1b-0.yarnrm.bigdata.svc.cluster.local,yarnrm1b-1.yarnrm.bigdata.svc.cluster.local spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES | http://yarnrm1b-0.yarnrm.bigdata.svc.cluster.local:8088/proxy/application_1693383838041_3359264,http://yarnrm1b-1.yarnrm.bigdata.svc.cluster.local:8088/proxy/application_1693383838041_3359264 spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.RM_HA_URLS | yarnrm1b-0.yarnrm.bigdata.svc.cluster.local:8088,yarnrm1b-1.yarnrm.bigdata.svc.cluster.local:8088 spark.plugins | io.glutenproject.GlutenPlugin spark.reducer.maxBlocksInFlightPerAddress | 1000 spark.reducer.maxReqsInFlight | 1000 spark.repl.class.outputDir | /data/data1/yarn/nm/usercache/hive/appcache/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000001/tmp/spark7852604390862141239 spark.repl.class.uri | spark://xxx:44823/classes spark.scheduler.mode | FIFO spark.serializer | org.apache.spark.serializer.KryoSerializer spark.shuffle.consolidateFiles | true spark.shuffle.io.maxRetries | 5 spark.shuffle.io.retryWait | 10 spark.shuffle.manager | org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager spark.shuffle.registration.maxAttempts | 5 spark.shuffle.registration.timeout | 120000 spark.shuffle.service.enabled | false spark.shuffle.useOldFetchProtocol | true spark.speculation | true spark.speculation.interval | 10000 spark.speculation.quantile | 0.95 spark.sql.adaptive.advisoryPartitionSizeInBytes | 134217728 spark.sql.adaptive.enabled | true spark.sql.adaptive.localShuffleReader.enabled | false spark.sql.adaptive.shuffle.targetPostShuffleInputSize | 268435456 spark.sql.autoBroadcastJoinThreshold | 20971520 spark.sql.broadcastTimeout | 1200 spark.sql.catalogImplementation | hive spark.sql.compatible.check.enabled | false spark.sql.extensions | io.glutenproject.GlutenSessionExtensions spark.sql.files.maxPartitionBytes | 268435456 spark.sql.files.openCostInBytes | 8388608 spark.sql.files.readParallelism | 1 spark.sql.hive.caseSensitiveInferenceMode | NEVER_INFER spark.sql.legacy.allowHashOnMapType | true spark.sql.legacy.timeParserPolicy | LEGACY spark.sql.mapKeyDedupPolicy | LAST_WIN spark.sql.orc.compression.codec | zlib spark.sql.parquet.fs.optimized.committer.optimization-enabled | false spark.sql.parquet.writeLegacyFormat | true spark.sql.shuffle.partitions | 400 spark.sql.sources.parallelPartitionDiscovery.parallelism | 10 spark.sql.storeAssignmentPolicy | LEGACY spark.storage.decommission.enabled | true spark.storage.decommission.fallbackStorage.path | s3a://bi.oppo.com/yarn-eks/bigdata-default/ spark.storage.decommission.rddBlocks.enabled | true spark.storage.decommission.shuffleBlocks.enabled | true spark.submit.deployMode | cluster spark.ui.filters | org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter spark.ui.port | 0 spark.yarn.am.extraJavaOptions | -Ddubbo.application.qos.enable=false -Dcom.amazonaws.services.s3.enableV4=true -Djava.net.preferIPv4Stack=true -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:MaxDirectMemorySize=2g -XX:+UseCompressedOops -XX:ParallelGCThreads=8 -XX:ConcGCThreads=4 -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:OnOutOfMemoryError="kill -9 %p" -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Xloggc:/gc.log
spark.yarn.am.memory | 2g
spark.yarn.am.waitTime | 600s
spark.yarn.app.container.log.dir | /data/data1/yarn/container-logs/application_1693383838041_3359264/container_e32_1693383838041_3359264_01_000001
spark.yarn.app.id | application_1693383838041_3359264
spark.yarn.archive | s3a:/xxxxapp/spark/spark3.4.2-gluten1.1.1-centos8-arm-20240523-1619.zip
spark.yarn.dist.archives | s3a://xxx/app/spark/sparkr/sparkr-20230509.zip#sparkr
spark.yarn.dist.files | file:///opt/apache-livy-0.7.1-incubating-SNAPSHOT-bin/conf/yarn/hive-site.xml
spark.yarn.historyServer.address | sparkhs:18080
spark.yarn.isPython | true
spark.yarn.maxAppAttempts | 1
spark.yarn.priority | 4
spark.yarn.queue | root.wireless_sg.daily
spark.yarn.secondary.jars | AdsUDF.2.0.0.jar,appstore_exposure_udf-1.0.jar,bdp_udf-1.0-SNAPSHOT.jar,hive_udf-1.0-jar-ip.jar,hive_udf-1.0-jar-position.jar,hive_udf-1.0.jar,hive_udfs-1.0.0.jar,universe_cdo_expose_obj_opt_format.jar,GeoIpParse.jar,kryo-shaded-4.0.2.jar,livy-api-0.7.0-incubating-SNAPSHOT.jar,livy-rsc-0.7.0-incubating-SNAPSHOT.jar,livy-thriftserver-session-0.7.0-incubating-SNAPSHOT.jar,minlog-1.3.0.jar,netty-all-4.1.47.Final.jar,objenesis-2.5.1.jar,commons-codec-1.9.jar,livy-client-common-0.7.0-incubating-SNAPSHOT.jar,livy-core_2.12-0.7.0-incubating-SNAPSHOT.jar,livy-repl_2.12-0.7.0-incubating-SNAPSHOT.jar,datanucleus-rdbms-4.1.19.jar,datanucleus-core-4.1.17.jar,datanucleus-api-jdo-4.2.4.jar
spark.yarn.submit.waitAppCompletion | false
System information
Velox System Info v0.0.2 Commit: 8a935d575aa0a38c3324f7ee98c87b576eb7ad70 CMake Version: 3.20.2 System: Linux-4.18.0-240.10.1.el8_3.aarch64 Arch: aarch64 C++ Compiler: /opt/rh/gcc-toolset-10/root/usr/bin/c++ C++ Compiler Version: 10.3.1 C Compiler: /opt/rh/gcc-toolset-10/root/usr/bin/cc C Compiler Version: 10.3.1 CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt
\nThe results will be copied to your clipboard if xclip is installed.
Relevant logs