facebookincubator / velox

A C++ vectorized database acceleration library aimed to optimizing query engines and data processing systems.
https://velox-lib.io/
Apache License 2.0
3.47k stars 1.14k forks source link

Flaky test S3FileSystemTest.invalidAccessKey #8471

Open duanmeng opened 9 months ago

duanmeng commented 9 months ago

Bug description

Flaky test S3FileSystemTest.invalidAccessKey

This was found by a PR CI test, https://app.circleci.com/pipelines/github/facebookincubator/velox/42841/workflows/c156d45b-7b47-4acc-b5f8-77dc893387b2/jobs/296842

Relevant logs

287: E0119 17:41:19.555821 96311 Exceptions.h:68] Line: ../../velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp:93, Function:initialize, Expression:  Failed to get metadata for S3 object due to: 'Network connection'. Path:'s3://dummy/foo.txt', SDK Error Type:99, HTTP Status Code:-1, S3 Service:'Unknown', Message:'curlCode: 7, Couldn't connect to server', RequestID:'', Source: RUNTIME, ErrorCode: INVALID_STATE
287: ../../velox/connectors/hive/storage_adapters/s3fs/tests/S3FileSystemTest.cpp:134: Failure
287: Value of: e.message().find("Failed to get metadata for S3 object due to: 'Access denied'. Path:'s3://dummy/foo.txt', SDK Error Type:15, HTTP Status Code:403, S3 Service:'MinIO', Message:'No response body.'") != std::string::npos
287:   Actual: false
287: Expected: true
287: Expected error message to contain 'Failed to get metadata for S3 object due to: 'Access denied'. Path:'s3://dummy/foo.txt', SDK Error Type:15, HTTP Status Code:403, S3 Service:'MinIO', Message:'No response body.'', but received 'Failed to get metadata for S3 object due to: 'Network connection'. Path:'s3://dummy/foo.txt', SDK Error Type:99, HTTP Status Code:-1, S3 Service:'Unknown', Message:'curlCode: 7, Couldn't connect to server', RequestID:'''.
287: I0119 17:41:19.557137 96311 TempDirectoryPath.cpp:31] TempDirectoryPath:: removing all files from/tmp/velox_test_S1xysT
287: [  FAILED  ] S3FileSystemTest.invalidAccessKey (27566 ms)
duanmeng commented 9 months ago

cc @mbasmanova @majetideepak

mbasmanova commented 9 months ago

@majetideepak Deepak, would you help take a look?