[C++] S3 stress tests can fail to delete a temporary directory

[bkietz]

Describe the enhancement requested

This seems to be causing spurious failures of the stress test, for example:

API: DeleteMultipleObjects(bucket=stress, multiObject=true, numberOfObjects=1000)
Time: 14:48:36 UTC 09/19/2023
DeploymentID: e8cea2aa-a72f-4074-90e9-0d3a974514d4
RequestID: 178654398B35E984
RemoteHost: 127.0.0.1
Host: 127.0.0.8:51443
UserAgent: aws-sdk-cpp/1.11.68 Windows/10.0.17763.4252 AMD64 MSVC/1929
Error: file access denied (cmd.StorageErr)
       6: internal\logger\logger.go:278:logger.LogIf()
       5: cmd\fs-v1-helpers.go:430:cmd.fsDeleteFile()
       4: cmd\fs-v1.go:1196:cmd.(*FSObjects).DeleteObject()
       3: cmd\fs-v1.go:1142:cmd.(*FSObjects).DeleteObjects()
       2: cmd\bucket-handlers.go:592:cmd.objectAPIHandlers.DeleteMultipleObjectsHandler()
       1: net\http\server.go:2047:http.HandlerFunc.ServeHTTP()
[FATAL] 2023-09-19 14:47:20.923 WinHttpSyncHttpClient [5648] Failed setting secure crypto protocols with error code: 87
[FATAL] 2023-09-19 14:47:21.524 WinHttpSyncHttpClient [5648] Failed setting secure crypto protocols with error code: 87
C:/projects/arrow/cpp/src/arrow/filesystem/s3fs_test.cc(865): error: Failed
'fs_->DeleteDirContents("stress")' failed with IOError: Got the following 1 errors when deleting objects in S3 bucket 'stress':
- key '1/508': Access Denied.

I'm not sure this is actually the issue; other tests can still pass even if deletion of a temporary directory succeeds:

[ RUN      ] TestS3FS.DeleteFile
[FATAL] 2023-09-19 14:48:38.075 WinHttpSyncHttpClient [5648] Failed setting secure crypto protocols with error code: 87
[FATAL] 2023-09-19 14:48:38.108 WinHttpSyncHttpClient [5648] Failed setting secure crypto protocols with error code: 87
C:/projects/arrow/cpp/src/arrow/status.cc:155: When trying to delete temporary directory: IOError: Cannot delete directory entry 'C:/Users/appveyor/AppData/Local/Temp/1/s3fs-test-mr25m6xu/.minio.sys': . Detail: [Windows error 145] The directory is not empty.
[       OK ] TestS3FS.DeleteFile (594 ms)

Component(s)

C++

[bkietz]

@mapleFU @pitrou

[pitrou]

I suppose this is only on Windows? Constraints on file deletion on Windows are annoying...

[bkietz]

See also https://github.com/apache/arrow/issues/30568

[bkietz]

Is the file deletion relevant to the test? Maybe we could "fix" it by just not deleting on Windows (maybe logging paths of files in case someone needs to delete them by hand)

[pitrou]

Is the file deletion relevant to the test? Maybe we could "fix" it by just not deleting on Windows (maybe logging paths of files in case someone needs to delete them by hand)

I would suggest retrying the DeleteDirContents call a number of times, with sleeps in between.