search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.15k stars 416 forks source link

[VL] Pool is null when using s3 cpp sdk to write data into s3 external table #6714

Open dcoliversun opened 1 month ago

dcoliversun commented 1 month ago

Backend

VL (Velox)

Bug description

Based gluten 468000c43efe7d54910431c7268768c4cb2d0410, when I test with velox to write data into s3 external table, I get the error as following:

ression: pool != nullptr , Source: RUNTIME, ErrorCode: INVALID_STATE
24/07/30 09:48:38 INFO CodecConfig: Compression: SNAPPY
E20240730 09:48:38.468577 93 Exceptions.h:67] Line: /gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp:229, Function:Impl, Expression: pool != nullptr , Source: RUNTIME, ErrorCode: INVALID_STATE
24/07/30 09:48:38 ERROR Utils: Aborting task
java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 229
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems11S3WriteFile4ImplC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS0_21MultiRoleArnsS3ClientEPNS0_6memory10MemoryPoolE
# 4 _ZN8facebook5velox11filesystems11S3WriteFileC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS0_21MultiRoleArnsS3ClientEPNS0_6memory10MemoryPoolE
# 5 _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6 _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7 _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8 _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9 _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007ff08d018767

Exception stack:

facebook::velox::filesystems::S3WriteFile::Impl::Impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, facebook::velox::MultiRoleArnsS3Client*, facebook::velox::memory::MemoryPool*)

facebook::velox::filesystems::S3WriteFile::S3WriteFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, facebook::velox::MultiRoleArnsS3Client*, facebook::velox::memory::MemoryPool*)

facebook::velox::filesystems::S3FileSystem::openFileForWrite(std::basic_string_view<char, std::char_traits<char> >, facebook::velox::filesystems::FileOptions const&)

facebook::velox::filesystems::s3WriteFileSinkGenerator(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, facebook::velox::dwio::common::FileSink::Options const&)

std::_Function_handler<std::unique_ptr<facebook::velox::dwio::common::FileSink, std::default_delete<facebook::velox::dwio::common::FileSink> > (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, facebook::velox::dwio::common::FileSink::Options const&), std::unique_ptr<facebook::velox::dwio::common::FileSink, std::default_delete<facebook::velox::dwio::common::FileSink> > (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, facebook::velox::dwio::common::FileSink::Options const&)>::_M_invoke(std::_Any_data const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, facebook::velox::dwio::common::FileSink::Options const&)

facebook::velox::dwio::common::FileSink::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, facebook::velox::dwio::common::FileSink::Options const&)

gluten::VeloxParquetDatasourceS3::initSink(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)

gluten::VeloxParquetDatasource::init(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)

org.apache.gluten.datasource.DatasourceJniWrapper#nativeInitDatasource

Spark sql is simple: 

create table tbl (a INT, b INT) using parquet location 's3a://<bucket>/<db>/<tbl>'
insert overwrite table values(1, 2);

Spark version

Spark-3.3.x

Spark configurations

bin/spark-shell \
--packages org.apache.hadoop:hadoop-aws:3.3.1 \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
--conf spark.driver.extraClassPath=/home/incubator-gluten/package/target/gluten-velox-bundle-spark3.3_2.12-centos_8_x86_64-1.2.0-SNAPSHOT.jar \
--conf spark.executor.extraClassPath=/home/incubator-gluten/package/target/gluten-velox-bundle-spark3.3_2.12-centos_8_x86_64-1.2.0-SNAPSHOT.jar \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
--conf spark.hadoop.fs.s3a.access.key=<accessKey>\
--conf spark.hadoop.fs.s3a.secret.key=<secretKey>\
--conf spark.hadoop.fs.s3a.endpoint=https://oss-ap-southeast-1.aliyuncs.com \
--conf spark.hadoop.fs.s3a.connection.ssl.enabled=true \
--conf spark.hadoop.fs.s3a.path.style.access=false \
--conf spark.eventLog.enabled=true \
--conf spark.gluten.velox.awsSdkLogLevel=DEBUG \
--conf spark.gluten.sql.native.writer.enabled=true

System information

Velox System Info v0.0.2
Commit: 468000c43efe7d54910431c7268768c4cb2d0410
CMake Version: 3.28.3
System: Linux-4.18.0-348.7.1.el8_5.x86_64
Arch: x86_64
CPU Name: Model name:          Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz
BIOS Model name:     pc-i440fx-2.1
C++ Compiler: /opt/rh/gcc-toolset-9/root/usr/bin/c++
C++ Compiler Version: 9.2.1
C Compiler: /opt/rh/gcc-toolset-9/root/usr/bin/cc
C Compiler Version: 9.2.1
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib64/python3.9/site-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt

Relevant logs

spark.sql("insert overwrite db.tbl values(2, 1)")
24/08/22 09:58:58 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
24/08/22 09:59:00 WARN AbstractS3ACommitterFactory: Using standard FileOutputCommitter to commit work. This is slow and potentially unsafe.
24/08/22 09:59:00 WARN AbstractS3ACommitterFactory: Using standard FileOutputCommitter to commit work. This is slow and potentially unsafe.
E20240822 09:59:06.508097 155687 Exceptions.h:67] Line: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp:227, Function:Impl, Expression: pool != nullptr , Source: RUNTIME, ErrorCode: INVALID_STATE
24/08/22 09:59:06 ERROR TaskResources: Task 3 failed by error: 
java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4  _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5  _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6  _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7  _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8  _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9  _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(Native Method)
        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(DatasourceJniWrapper.java:49)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter(VeloxFormatWriterInjects.scala:70)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter$(VeloxFormatWriterInjects.scala:43)
        at org.apache.spark.sql.execution.datasources.velox.VeloxParquetWriterInjects.createOutputWriter(VeloxParquetWriterInjects.scala:26)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:100)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:163)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:150)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:417)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:351)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
24/08/22 09:59:06 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4  _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5  _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6  _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7  _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8  _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9  _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(Native Method)
        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(DatasourceJniWrapper.java:49)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter(VeloxFormatWriterInjects.scala:70)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter$(VeloxFormatWriterInjects.scala:43)
        at org.apache.spark.sql.execution.datasources.velox.VeloxParquetWriterInjects.createOutputWriter(VeloxParquetWriterInjects.scala:26)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:100)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:163)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:150)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:417)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:351)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
24/08/22 09:59:06 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 3) (iZt4n8he1ls2xewpuxsh4yZ executor driver): java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4  _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5  _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6  _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7  _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8  _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9  _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(Native Method)
        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(DatasourceJniWrapper.java:49)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter(VeloxFormatWriterInjects.scala:70)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter$(VeloxFormatWriterInjects.scala:43)
        at org.apache.spark.sql.execution.datasources.velox.VeloxParquetWriterInjects.createOutputWriter(VeloxParquetWriterInjects.scala:26)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:100)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:163)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:150)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:417)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:351)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

24/08/22 09:59:06 ERROR TaskSetManager: Task 0 in stage 2.0 failed 1 times; aborting job
24/08/22 09:59:06 ERROR FileFormatWriter: Aborting job a9b92470-cf6e-4382-b32c-25323810b4bd.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3) (iZt4n8he1ls2xewpuxsh4yZ executor driver): java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4  _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5  _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6  _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7  _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8  _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9  _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(Native Method)
        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(DatasourceJniWrapper.java:49)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter(VeloxFormatWriterInjects.scala:70)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter$(VeloxFormatWriterInjects.scala:43)
        at org.apache.spark.sql.execution.datasources.velox.VeloxParquetWriterInjects.createOutputWriter(VeloxParquetWriterInjects.scala:26)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:100)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:163)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:150)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:417)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:351)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2228)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:340)
        at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:195)
        at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
        at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
        at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
        at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
        at $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:23)
        at $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)
        at $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:29)
        at $line17.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:31)
        at $line17.$read$$iw$$iw$$iw$$iw.<init>(<console>:33)
        at $line17.$read$$iw$$iw$$iw.<init>(<console>:35)
        at $line17.$read$$iw$$iw.<init>(<console>:37)
        at $line17.$read$$iw.<init>(<console>:39)
        at $line17.$read.<init>(<console>:41)
        at $line17.$read$.<init>(<console>:45)
        at $line17.$read$.<clinit>(<console>)
        at $line17.$eval$.$print$lzycompute(<console>:7)
        at $line17.$eval$.$print(<console>:6)
        at $line17.$eval.$print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
        at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
        at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
        at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
        at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
        at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
        at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
        at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:865)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:733)
        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:435)
        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:456)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:239)
        at org.apache.spark.repl.Main$.doMain(Main.scala:78)
        at org.apache.spark.repl.Main$.main(Main.scala:58)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4  _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5  _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6  _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7  _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8  _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9  _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(Native Method)
        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(DatasourceJniWrapper.java:49)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter(VeloxFormatWriterInjects.scala:70)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter$(VeloxFormatWriterInjects.scala:43)
        at org.apache.spark.sql.execution.datasources.velox.VeloxParquetWriterInjects.createOutputWriter(VeloxParquetWriterInjects.scala:26)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:100)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:163)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:150)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:417)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:351)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
org.apache.spark.SparkException: Job aborted.
  at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:651)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:376)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:195)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
  at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
  at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
  at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
  at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
  ... 47 elided
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3) (iZt4n8he1ls2xewpuxsh4yZ executor driver): java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4  _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5  _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6  _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7  _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8  _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9  _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(Native Method)
        at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(DatasourceJniWrapper.java:49)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter(VeloxFormatWriterInjects.scala:70)
        at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter$(VeloxFormatWriterInjects.scala:43)
        at org.apache.spark.sql.execution.datasources.velox.VeloxParquetWriterInjects.createOutputWriter(VeloxParquetWriterInjects.scala:26)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:100)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:163)
        at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:150)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:417)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:351)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
  at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)
  at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)
  at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)
  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)
  at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)
  at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)
  at scala.Option.foreach(Option.scala:407)
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2228)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:340)
  ... 78 more
Caused by: java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4  _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5  _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6  _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7  _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8  _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9  _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

  at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(Native Method)
  at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(DatasourceJniWrapper.java:49)
  at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter(VeloxFormatWriterInjects.scala:70)
  at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter$(VeloxFormatWriterInjects.scala:43)
  at org.apache.spark.sql.execution.datasources.velox.VeloxParquetWriterInjects.createOutputWriter(VeloxParquetWriterInjects.scala:26)
  at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:100)
  at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:163)
  at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:150)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:417)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:351)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:136)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
dcoliversun commented 1 month ago

cc @zhztheplayer Could you please check it? Thanks

FelixYBW commented 1 month ago

I didn't meet the issue. Is it a new one? Did you run before?

FelixYBW commented 1 month ago

Can you reproduce the stage using microbenchmark? Looks it's a Velox issue.

dcoliversun commented 1 month ago

@FelixYBW Sure, I will try

FelixYBW commented 1 month ago

I can't reproduce this bug from my environment. You may print out the pool pointer in the call chain and see where it's lost. Does it happen when the task finish?

dcoliversun commented 1 month ago

@FelixYBW I can reproduce this. Update info in description.

dcoliversun commented 1 month ago

Same exception with v1.2.0-rc3. 

Velox System Info v0.0.2
Commit: c82af60a635fb52bf1314cd831f5915dd37d910d
CMake Version: 3.28.3
System: Linux-4.18.0-348.7.1.el8_5.x86_64
Arch: x86_64
CPU Name: Model name:          Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz
BIOS Model name:     pc-i440fx-2.1
C++ Compiler: /opt/rh/gcc-toolset-9/root/usr/bin/c++
C++ Compiler Version: 9.2.1
C Compiler: /opt/rh/gcc-toolset-9/root/usr/bin/cc
C Compiler Version: 9.2.1
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib64/python3.9/site-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt