Open xingnailu opened 10 months ago
I have similar exception but data is stored on Alibaba OSS. S3 Storage Adapters support oss scheme[1]
Exception info is
Caused by: io.glutenproject.exception.GlutenException: java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to get metadata for S3 object due to: 'Resource not found'. Path:'s3://henghzhen-test-hangzhou/db/t1/b=1/c=10/part-00000-d4940ed1-7f70-44f5-bbb0-65ae29f325f1.c000.snappy.parquet', SDK Error Type:16, HTTP Status Code:404, S3 Service:'AmazonS3', Message:'No response body.', RequestID:'2VQQRSWNX8QQGNNY'
Retriable: False
Context: Split [Hive: s3a://henghzhen-test-hangzhou/db/t1/b=1/c=10/part-00000-d4940ed1-7f70-44f5-bbb0-65ae29f325f1.c000.snappy.parquet 0 - 443] Task Gluten_Stage_0_TID_0
Top-Level Context: Same as context.
Function: initialize
File: ../../velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 93
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox12_GLOBAL__N_110S3ReadFile10initializeEv
# 4 _ZN8facebook5velox11filesystems12S3FileSystem15openFileForReadESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 5 _ZN8facebook5velox19FileHandleGeneratorclERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
# 6 _ZN8facebook5velox13CachedFactoryINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt10shared_ptrINS0_10FileHandleEENS0_19FileHandleGeneratorEE8generateERKS7_
# 7 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 8 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 9 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 10 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 11 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 12 _ZN6gluten24WholeStageResultIterator4nextEv
# 13 Java_io_glutenproject_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 14 0x00007f8c75018427
[1] https://facebookincubator.github.io/velox/develop/connectors.html?highlight=oss
Bug description
Bug description I built gluten+velox using branch-1.1, submitted a tpch query using spark-shell, and the data was stored in s3. However, the following error occurred during execution:
Reason: Failed to get metadata for S3 object due to: 'Unknown error'. Path:'s3://xxxxxxx/user/hive/warehouse/tpch_orc.db/customer/part-00027-31ef1f3c-5b27-4f6c-aef4-7f77f7749873-c000.snappy.orc', SDK Error Type:100, HTTP Status Code:400, S3 Service:'AmazonS3', Message:'No response body.', RequestID:'KC5WQZ78QWKQ9BFX'"
But I can use gluten tag v1.0.0 version to execute normally.
@majetideepak
System information
System information build branch-1.1 system info:
Velox System Info v0.0.2 Commit: https://github.com/facebookincubator/velox/commit/bbd65c4109fc11d4021334aff817ff384eab7b88 CMake Version: 3.16.3 System: Linux-5.15.0-91-generic Arch: x86_64 C++ Compiler: /bin/c++ C++ Compiler Version: 9.4.0 C Compiler: /bin/cc C Compiler Version: 9.4.0 CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt
run on aws eks
Relevant logs