search

kwai / blaze

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.
Apache License 2.0
1.27k stars 119 forks source link

run spark+tpcds 1T with blaze, execution error happened #634

Open shaileneF opened 3 days ago

shaileneF commented 3 days ago

env：

host
os：CentOS Linux release 8.2.2004 (Core)
kernel：4.18.0-193.el8.x86_64
container：
os：CentOS Linux release 7.4.1708 (Core)
kernel：4.18.0-193.el8.x86_64
cpu cores：128
spark: 3.4.3

spark-default.conf:

spark.master local
spark.eventLog.enabled true
spark.eventLog.dir /home/blaze-test/spark-eventlog
spark.history.fs.logDirectory /home/blaze-test/spark-eventlog
spark.externalBlockStore.url.list /home/blaze-test/platform
spark.driver.extraJavaOptions -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/blaze-test/spark/ -Djava.io.tmpdir=/home/blaze-test/tmp -Dlog4j2.formatMsgNoLookups=true
spark.local.dir /home/blaze-test/localdir
spark.shuffle.service.enabled true
spark.shuffle.service.port 7337
spark.driver.memory 10g
spark.driver.memoryOverhead 4096
spark.executor.instances 12
spark.dynamicallocation.maxExecutors 12
spark.executor.cores 8
spark.io.compression.codec zstd
spark.sql.parquet.compression.codec zstd
spark.executor.memory 10g
spark.executor.memoryOverhead 4096
spark.blaze.enable true
spark.sql.extensions org.apache.spark.sql.blaze.BlazeSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleManager
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 40g

description ：

hello，I run spark+blaze+tpcds in local model. There is no errors with 1GB and 10GB. However, with 100GB, a few queries resulted in errors, and with 1TB, more than half of the queries encountered errors.

errors:

output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Project] error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Project] error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Project] error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Project] error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
(+6895.714s) [ERROR] Blaze - [partition=1] native execution panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
24/10/24 21:34:37 WARN TaskSetManager: Lost task 1.0 in stage 3403.0 (TID 31521) (22aa3cf9e3ca executor driver): java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
24/10/24 21:34:37 INFO DAGScheduler: ShuffleMapStage 3403 (main at NativeMethodAccessorImpl.java:0) failed in 0.136 s due to Job aborted due to stage failure: Task 1 in stage 3403.0 failed 1 times, most recent failure: Lost task 1.0 in stage 3403.0 (TID 31521) (22aa3cf9e3ca executor driver): java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
case: q96, round: 2, error: Job aborted due to stage failure: Task 1 in stage 3403.0 failed 1 times, most recent failure: Lost task 1.0 in stage 3403.0 (TID 31521) (22aa3cf9e3ca executor driver): java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 3403.0 failed 1 times, most recent failure: Lost task 1.0 in stage 3403.0 (TID 31521) (22aa3cf9e3ca executor driver): java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
Caused by: java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] canceled due to task finished/killed
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] canceled due to task finished/killed
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] canceled due to task finished/killed
output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] canceled due to task finished/killed
richox commented 3 days ago

this should have been fixed in #616 , you can try compiling with the latest code.

wangfann commented 1 day ago

I had the same error problem, then I compiled with the latest master branch code and still got the error, the error message changed to Caused by: java.lang.NullPointerException at org.apache.spark.util.SparkErrorUtils.tryWithResource(SparkErrorUtils.scala:48) at org.apache.spark.util.SparkErrorUtils.tryWithResource$(SparkErrorUtils.scala:46) at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:94) at org.apache.spark.sql.blaze.BlazeCallNativeWrapper$.loadLibBlaze(BlazeCallNativeWrapper.scala:205) at org.apache.spark.sql.blaze.BlazeCallNativeWrapper$.lazyInitNative$lzycompute(BlazeCallNativeWrapper.scala:190) at org.apache.spark.sql.blaze.BlazeCallNativeWrapper$.lazyInitNative(BlazeCallNativeWrapper.scala:183) at org.apache.spark.sql.blaze.BlazeCallNativeWrapper$.initNative(BlazeCallNativeWrapper.scala:180) at org.apache.spark.sql.blaze.BlazeCallNativeWrapper.(BlazeCallNativeWrapper.scala:55) at org.apache.spark.sql.blaze.NativeHelper$.executeNativePlan(NativeHelper.scala:89) at org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleWriterBase.nativeShuffleWrite(BlazeShuffleWriterBase.scala:80) at org.apache.spark.sql.execution.blaze.plan.NativeShuffleExchangeExec$$anon$1.write(NativeShuffleExchangeExec.scala:158) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) My spark version is 3.5.2，and my sql is "select tb1.cs_sold_date_sk, count() from (select from tpcds_test.catalog_sales_tmp1 limit 1000) tb1 left join tpcds_test.catalog_sales_tmp3 tb2 on tb1.cs_sold_date_sk = tb2.cs_sold_date_sk and tb1.cs_sold_time_sk = tb2.cs_sold_time_sk group by tb1.cs_sold_date_sk"

shaileneF commented 1 day ago

this should have been fixed in #616 , you can try compiling with the latest code.

thank you，I am trying.

shaileneF commented 6 hours ago

this should have been fixed in #616 , you can try compiling with the latest code.

I got some compiling error with the latest main branch code. 🙏

rust version: rustc 1.82.0 (f6e511eec 2024-10-15) cargo version: cargo 1.82.0 (8f40fc59f 2024-08-21)

[ERROR] [main] find: 'target/release/libblaze.so': No such file or directory
[INFO] [main] old build-checksum: cat: ./.build-checksum_release-so.cache: No such file or directory
[INFO] [main] new build-checksum: 2e1f0422f62271e27a5f4a2b923d44e1  -
[INFO] [main] Running cargo fix...
[ERROR] [main]  Downloading crates ...
[ERROR] [main]   Downloaded lzokay-native v0.1.0
[ERROR] [main]    Compiling proc-macro2 v1.0.87
[ERROR] [main]    Compiling unicode-ident v1.0.13
[ERROR] [main]    Compiling libc v0.2.159
[ERROR] [main]    Compiling autocfg v1.4.0
[ERROR] [main]     Checking cfg-if v1.0.0
[ERROR] [main]    Compiling libm v0.2.8
[ERROR] [main]    Compiling shlex v1.3.0
[ERROR] [main]    Compiling version_check v0.9.5
[ERROR] [main]     Checking bytes v1.7.2
[ERROR] [main]    Compiling regex-syntax v0.8.5
[ERROR] [main]     Checking byteorder v1.5.0
[ERROR] [main]    Compiling serde v1.0.210
[ERROR] [main]    Compiling siphasher v0.3.11
[ERROR] [main]    Compiling rand_core v0.6.4
[ERROR] [main]    Compiling pkg-config v0.3.31
[ERROR] [main]     Checking once_cell v1.20.2
[ERROR] [main]     Checking memchr v2.7.4
[ERROR] [main]    Compiling heck v0.5.0
[ERROR] [main]     Checking bitflags v2.6.0
[ERROR] [main]     Checking iana-time-zone v0.1.61
[ERROR] [main]     Checking static_assertions v1.1.0
[ERROR] [main]     Checking allocator-api2 v0.2.18
[ERROR] [main]    Compiling rustversion v1.0.17
[ERROR] [main]     Checking pin-project-lite v0.2.14
[ERROR] [main]     Checking ryu v1.0.18
[ERROR] [main]    Compiling semver v1.0.23
[ERROR] [main]     Checking futures-core v0.3.31
[ERROR] [main]     Checking futures-sink v0.3.31
[ERROR] [main]     Checking unicode-width v0.1.14
[ERROR] [main]    Compiling parking_lot_core v0.9.10
[ERROR] [main]     Checking futures-io v0.3.31
[ERROR] [main]     Checking base64 v0.22.1
[ERROR] [main]     Checking itoa v1.0.11
[ERROR] [main]     Checking pin-utils v0.1.0
[ERROR] [main]     Checking smallvec v1.13.2
[ERROR] [main]     Checking tinyvec_macros v0.1.1
[ERROR] [main]     Checking scopeguard v1.2.0
[ERROR] [main]     Checking futures-task v0.3.31
[ERROR] [main]    Compiling serde_json v1.0.96 (https://github.com/blaze-init/json?branch=v1.0.96-blaze#95fa6cba)
[ERROR] [main]     Checking bitflags v1.3.2
[ERROR] [main]     Checking log v0.4.22
[ERROR] [main]    Compiling zstd-safe v7.2.1
[ERROR] [main]     Checking equivalent v1.0.1
[ERROR] [main]     Checking hashbrown v0.15.0
[ERROR] [main]     Checking adler2 v2.0.0
[ERROR] [main]     Checking alloc-no-stdlib v2.0.4
[ERROR] [main]    Compiling snap v1.1.1
[ERROR] [main]     Checking unicode-bidi v0.3.17
[ERROR] [main]     Checking percent-encoding v2.3.1
[ERROR] [main]     Checking same-file v1.0.6
[ERROR] [main]     Checking lazy_static v1.5.0
[ERROR] [main]    Compiling paste v1.0.15
[ERROR] [main]     Checking either v1.13.0
[ERROR] [main]     Checking integer-encoding v3.0.4
[ERROR] [main]     Checking humantime v2.1.0
[ERROR] [main]    Compiling rustix v0.38.37
[ERROR] [main]    Compiling seq-macro v0.3.5
[ERROR] [main]    Compiling crossbeam-utils v0.8.20
[ERROR] [main]    Compiling typenum v1.17.0
[ERROR] [main]     Checking linux-raw-sys v0.4.14
[ERROR] [main]     Checking fastrand v2.1.1
[ERROR] [main]     Checking fixedbitset v0.4.2
[ERROR] [main]     Checking hex v0.4.3
[ERROR] [main]     Checking subtle v2.6.1
[ERROR] [main]    Compiling thiserror v1.0.64
[ERROR] [main]    Compiling anyhow v1.0.89
[ERROR] [main]    Compiling syn v1.0.109
[ERROR] [main]     Checking constant_time_eq v0.3.1
[ERROR] [main]     Checking cpufeatures v0.2.14
[ERROR] [main]     Checking arrayref v0.3.9
[ERROR] [main]     Checking arrayvec v0.7.6
[ERROR] [main]     Checking unicode-segmentation v1.12.0
[ERROR] [main]    Compiling radium v0.7.0
[ERROR] [main]     Checking jni-sys v0.3.0
[ERROR] [main]     Checking glob v0.3.1
[ERROR] [main]     Checking cesu8 v1.1.0
[ERROR] [main]     Checking tap v1.0.1
[ERROR] [main]     Checking funty v2.0.0
[ERROR] [main]     Checking radsort v0.1.1
[ERROR] [main]     Checking unchecked-index v0.2.2
[ERROR] [main]     Checking simdutf8 v0.1.5
[ERROR] [main]    Compiling prettyplease v0.2.22
[ERROR] [main]    Compiling zstd-safe v6.0.6
[ERROR] [main]     Checking bumpalo v3.16.0
[ERROR] [main]    Compiling multimap v0.10.0
[ERROR] [main]    Compiling gxhash v3.4.1
[ERROR] [main]     Checking fallible-streaming-iterator v0.1.9
[ERROR] [main]     Checking panic-message v0.3.0
[ERROR] [main]     Checking count-write v0.1.0
[ERROR] [main]     Checking bytesize v1.3.0
[ERROR] [main]    Compiling phf_shared v0.11.2
[ERROR] [main]    Compiling rand v0.8.5
[ERROR] [main]     Checking lexical-util v0.8.5
[ERROR] [main]     Checking futures-channel v0.3.31
[ERROR] [main]     Checking twox-hash v1.6.3
[ERROR] [main]     Checking tinyvec v1.8.0
[ERROR] [main]     Checking miniz_oxide v0.8.0
[ERROR] [main]     Checking crc32fast v1.4.2
[ERROR] [main]     Checking alloc-stdlib v0.2.2
[ERROR] [main]     Checking form_urlencoded v1.2.1
[ERROR] [main]     Checking tracing-core v0.1.32
[ERROR] [main]     Checking walkdir v2.5.0
[ERROR] [main]     Checking wyz v0.5.1
[ERROR] [main]     Checking itertools v0.13.0
[ERROR] [main]    Compiling itertools v0.12.1
[ERROR] [main]     Checking brotli-decompressor v4.0.1
[ERROR] [main]     Checking phf v0.11.2
[ERROR] [main]     Checking lz4_flex v0.11.3
[ERROR] [main]     Checking aho-corasick v1.1.3
[ERROR] [main]     Checking csv-core v0.1.11
[ERROR] [main]     Checking combine v4.6.7
[ERROR] [main]     Checking indexmap v2.6.0
[ERROR] [main]     Checking flate2 v1.0.34
[ERROR] [main]     Checking lexical-parse-integer v0.8.6
[ERROR] [main]     Checking lexical-write-integer v0.8.5
[ERROR] [main]    Compiling ahash v0.8.11
[ERROR] [main]    Compiling generic-array v0.14.7
[ERROR] [main]    Compiling phf_generator v0.11.2
[ERROR] [main]     Checking lexical-parse-float v0.8.5
[ERROR] [main]     Checking lexical-write-float v0.8.5
[ERROR] [main]    Compiling num-traits v0.2.19
[ERROR] [main]    Compiling slab v0.4.9
[ERROR] [main]    Compiling lock_api v0.4.12
[ERROR] [main]    Compiling bigdecimal v0.4.5
[ERROR] [main]    Compiling phf_codegen v0.11.2
[ERROR] [main]     Checking unicode-normalization v0.1.24
[ERROR] [main]    Compiling jni v0.20.0
[ERROR] [main]     Checking brotli v6.0.0
[ERROR] [main]     Checking petgraph v0.6.5
[ERROR] [main]     Checking lexical-core v0.8.5
[ERROR] [main]     Checking bitvec v1.0.1
[ERROR] [main]     Checking idna v0.5.0
[ERROR] [main]    Compiling regex-automata v0.4.8
[ERROR] [main]    Compiling rustc_version v0.4.1
[ERROR] [main]    Compiling quote v1.0.37
[ERROR] [main]    Compiling jobserver v0.1.32
[ERROR] [main]    Compiling flatbuffers v24.3.25
[ERROR] [main]     Checking getrandom v0.2.15
[ERROR] [main]     Checking num_cpus v1.16.0
[ERROR] [main]    Compiling syn v2.0.79
[ERROR] [main]     Checking threadpool v1.8.1
[ERROR] [main]     Checking uuid v1.10.0
[ERROR] [main]     Checking url v2.5.2
[ERROR] [main]     Checking parking_lot v0.12.3
[ERROR] [main]    Compiling cc v1.1.28
[ERROR] [main]     Checking num-integer v0.1.46
[ERROR] [main]     Checking num-complex v0.4.6
[ERROR] [main]     Checking half v2.4.1
[ERROR] [main]     Checking chrono v0.4.38
[ERROR] [main]     Checking atoi v2.0.0
[ERROR] [main]     Checking ordered-float v2.10.1
[ERROR] [main]     Checking thrift v0.17.0
[ERROR] [main]     Checking num-bigint v0.4.6
[ERROR] [main]     Checking num-iter v0.1.45
[ERROR] [main]     Checking crypto-common v0.1.6
[ERROR] [main]     Checking block-buffer v0.10.4
[ERROR] [main]     Checking digest v0.10.7
[ERROR] [main]     Checking tempfile v3.13.0
[ERROR] [main]     Checking md-5 v0.10.6
[ERROR] [main]     Checking sha2 v0.10.8
[ERROR] [main]     Checking blake2 v0.10.6
[ERROR] [main]    Compiling zstd-sys v2.0.13+zstd.1.5.6
[ERROR] [main]    Compiling lzma-sys v0.1.20
[ERROR] [main]    Compiling blake3 v1.5.4
[ERROR] [main]    Compiling bzip2-sys v0.1.11+1.0.8
[ERROR] [main]    Compiling jemalloc-sys v0.5.4+5.3.0-patched
[ERROR] [main]    Compiling regex v1.11.0
[ERROR] [main]     Checking num-rational v0.4.2
[ERROR] [main]    Compiling parse-zoneinfo v0.3.1
[ERROR] [main]     Checking num v0.4.3
[ERROR] [main]     Checking arrow-buffer v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]    Compiling chrono-tz-build v0.3.0
[ERROR] [main]     Checking bzip2 v0.4.4
[ERROR] [main]     Checking xz2 v0.1.7
[ERROR] [main]    Compiling chrono-tz v0.9.0
[ERROR] [main]    Compiling ptr_meta_derive v0.2.0
[ERROR] [main]    Compiling derivative v2.2.0
[ERROR] [main]    Compiling serde_derive v1.0.210
[ERROR] [main]    Compiling zerocopy-derive v0.7.35
[ERROR] [main]    Compiling strum_macros v0.26.4
[ERROR] [main]    Compiling tokio-macros v2.4.0
[ERROR] [main]    Compiling futures-macro v0.3.31
[ERROR] [main]    Compiling snafu-derive v0.8.5
[ERROR] [main]    Compiling async-trait v0.1.83
[ERROR] [main]    Compiling tracing-attributes v0.1.27
[ERROR] [main]    Compiling sqlparser_derive v0.2.2
[ERROR] [main]    Compiling thiserror-impl v1.0.64
[ERROR] [main]    Compiling prost-derive v0.13.3
[ERROR] [main]    Compiling prost-derive v0.12.6
[ERROR] [main]     Checking ptr_meta v0.2.0
[ERROR] [main]     Checking slimmer_box v0.6.5
[ERROR] [main]     Checking zstd v0.13.2
[ERROR] [main]     Checking zstd v0.12.4
[ERROR] [main]     Checking sqlparser v0.50.0
[ERROR] [main]     Checking tokio v1.40.0
[ERROR] [main]     Checking futures-util v0.3.31
[ERROR] [main]     Checking zerocopy v0.7.35
[ERROR] [main]     Checking lzokay-native v0.1.0
[ERROR] [main]     Checking tracing v0.1.40
[ERROR] [main]     Checking ppv-lite86 v0.2.20
[ERROR] [main]     Checking prost v0.12.6
[ERROR] [main]    Compiling prost v0.13.3
[ERROR] [main]     Checking hashbrown v0.14.5
[ERROR] [main]     Checking strum v0.26.3
[ERROR] [main]     Checking snafu v0.8.5
[ERROR] [main]     Checking comfy-table v7.1.1
[ERROR] [main]     Checking rand_chacha v0.3.1
[ERROR] [main]    Compiling prost-types v0.13.3
[ERROR] [main]     Checking dashmap v6.1.0
[ERROR] [main]    Compiling prost-build v0.13.3
[ERROR] [main]     Checking futures-executor v0.3.31
[ERROR] [main]    Compiling tonic-build v0.12.3
[ERROR] [main]     Checking futures v0.3.31
[ERROR] [main]     Checking object_store v0.11.0
[ERROR] [main]     Checking datafusion-common-runtime v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking async-compression v0.4.13
[ERROR] [main]     Checking tokio-util v0.7.12
[ERROR] [main]    Compiling blaze-serde v0.1.0 (/home/blaze/native-engine/blaze-serde)
[ERROR] [main]     Checking arrow-schema v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking csv v1.3.0
[ERROR] [main]     Checking faststr v0.2.23
[ERROR] [main]     Checking sonic-rs v0.3.13
[ERROR] [main]     Checking arrow-data v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-array v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-select v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-arith v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-row v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-cast v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-ord v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-string v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-ipc v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-csv v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-json v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking parquet v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking orc-rust v0.3.1 (https://github.com/blaze-init/datafusion-orc.git?rev=c54bfb5#c54bfb5b)
[ERROR] [main]     Checking datafusion-common v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-expr-common v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-physical-expr-common v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions-aggregate-common v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-expr v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-execution v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-sql v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions-window v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-physical-expr v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions-aggregate v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-optimizer v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-physical-plan v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions-nested v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking jemallocator v0.5.4
[ERROR] [main]     Checking datafusion-physical-optimizer v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-catalog v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking blaze-jni-bridge v0.1.0 (/home/blaze/native-engine/blaze-jni-bridge)
[ERROR] [main]     Checking datafusion-ext-commons v0.1.0 (/home/blaze/native-engine/datafusion-ext-commons)
[ERROR] [main] error[E0554]: `#![feature]` may not be used on the stable release channel
[ERROR] [main]   --> native-engine/datafusion-ext-commons/src/lib.rs:15:1
[ERROR] [main]    |
[ERROR] [main] 15 | #![feature(new_uninit)]
[ERROR] [main]    | ^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] [main]
[ERROR] [main] error[E0554]: `#![feature]` may not be used on the stable release channel
[ERROR] [main]   --> native-engine/datafusion-ext-commons/src/lib.rs:16:1
[ERROR] [main]    |
[ERROR] [main] 16 | #![feature(slice_swap_unchecked)]
[ERROR] [main]    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] [main]
[ERROR] [main] error[E0554]: `#![feature]` may not be used on the stable release channel
[ERROR] [main]   --> native-engine/datafusion-ext-commons/src/lib.rs:17:1
[ERROR] [main]    |
[ERROR] [main] 17 | #![feature(vec_into_raw_parts)]
[ERROR] [main]    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] [main]
[ERROR] [main] warning: the feature `new_uninit` has been stable since 1.82.0 and no longer requires an attribute to enable
[ERROR] [main]   --> native-engine/datafusion-ext-commons/src/lib.rs:15:12
[ERROR] [main]    |
[ERROR] [main] 15 | #![feature(new_uninit)]
[ERROR] [main]    |            ^^^^^^^^^^
[ERROR] [main]    |
[ERROR] [main]    = note: `#[warn(stable_features)]` on by default
[ERROR] [main]
[ERROR] [main] For more information about this error, try `rustc --explain E0554`.
[ERROR] [main] warning: `datafusion-ext-commons` (lib) generated 1 warning
[ERROR] [main] error: could not compile `datafusion-ext-commons` (lib) due to 3 previous errors; 1 warning emitted
[ERROR] [main] warning: build failed, waiting for other jobs to finish...
[ERROR] [main] warning: `datafusion-ext-commons` (lib test) generated 1 warning (1 duplicate)
[ERROR] [main] error: could not compile `datafusion-ext-commons` (lib test) due to 3 previous errors; 1 warning emitted
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 101 (Exit value: 101)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:982)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:929)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:452)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for blaze-engine 4.0.0-SNAPSHOT:
[INFO]
[INFO] blaze-engine ....................................... SUCCESS [  1.228 s]
[INFO] proto .............................................. SUCCESS [  2.456 s]
[INFO] spark-extension .................................... SUCCESS [  8.056 s]
[INFO] spark-extension-shims-spark3 ....................... SUCCESS [  2.189 s]
[INFO] assembly ........................................... FAILURE [01:36 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------