DeleteObject command giving NotImplemented error

[pballart]

Describe the bug I have an S3 compatible bucket in DigitalOcean Storage which I can access and use with other tools like Cyberduck. I implemented upload and deletion of an object using soto in my Vapor service. The problem I have is that whereas the PutObject works fine, the DeleteObject is giving me an error.

To Reproduce Steps to reproduce the behavior:

1. Create a bucket in DigitalOcean Storage
2. Upload a file
3. Try to delete the file

Expected behavior The file is deleted

Actual result The request fails with Unhandled error, code: notImplemented My guess is that there is some header in the request that shouldn't be there and it can't process. What bothers me is that using the same S3 bucket with other SDKs or even with a direct cURL works fine so I guess it must be something related to soto 🤔

Setup (please complete the following information):

• OS: MacOS 12.3.1
• Version of soto: 6.0.0
• Authentication mechanism: hard-coded credentials

Additional context I ran the command with the logging middleware:

Request:
  DeleteObject
  DELETE https://region.digitaloceanspaces.com/bucket/file.txt?x-id=DeleteObject
  Headers: [
    user-agent : Soto/6.0
    content-type : application/octet-stream
  ]
  Body: empty
Response:
  Status : 501
  Headers: [
    content-length : 193
    x-amz-request-id : tx00000000000000805c1d1-0062bf3992-51f730ea-fra1b
    accept-ranges : bytes
    content-type : application/xml
    date : Fri, 01 Jul 2022 18:14:42 GMT
    cache-control : max-age=60
    strict-transport-security : max-age=15552000; includeSubDomains; preload
  ]
  Body: 
  <Error><Code>NotImplemented</Code><RequestId>tx00000000000000805c1d1-0062bf3992-51f730ea-fra1b</RequestId><HostId>redacted-host-id</HostId></Error>

The code I'm using is pretty simple:

let deleteObjectRequest = S3.DeleteObjectRequest(
            bucket: "bucket",
            key: "file.jpg"
        )
        _ = try await s3.deleteObject(deleteObjectRequest)

and doing the following cURL works fine:

curl -X DELETE  \
-H "user-agent: Soto/6.0" \
-H "content-type: application/octet-stream" \
-H "host: region.digitaloceanspaces.com" \
-H "x-amz-date: 20220701T173944Z" \
-H "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
-H "Authorization: AWS4-HMAC-SHA256 Credential=REDACTED" \
"https://region.digitaloceanspaces.com/bucket/file.txt?x-id=DeleteObject"

[adam-fowler]

Not sure how the curl call works but Soto doesn't. The full HTTP request Soto sends includes all the same headers you sent. I thought possibly the content-type header might be the issue given there is no content. I verified aws-cli doesn't send the content type with empty requests.

What signed headers do you include in the authorisation field?

[pballart]

In the authorisation header I have this content, which is the same that soto sends:

AWS4-HMAC-SHA256 Credential=XXXXXX, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=572e50fb4ce317951a2cb23d42ba2363612d0bd5689153dd305fbdeed33b010d

What boggles me is that I literally print the request before being sent, copy paste the headers in the curl command and it works there. So far we know that:

1. The bucket works fine since I can delete from other clients
2. The curl with the same headers works
3. The NotImplemented error according to the docs refers to an unrecognised header being sent (but it might be something else)
4. My implementation was working some months ago and suddenly stopped working. Going back to an older version of soto doesn't fix it
5. Another source of the bug could be the AsyncHTTP swift library that maybe adds some headers? 🤔

[adam-fowler]

I'm currently removing the content-type header as it is a discrepancy between the aws-cli and Soto. I guess you can verify once that is committed

[adam-fowler]

SotoCore has a branch empty-content-type. If you have your own version of Soto you could set its Soto-core dependency to use that branch.

[pballart]

I tried both the branch empty-content-type and also the remove-headers-from-signature and still the same problem. 😞 Also the curl keeps working... super weird.

[adam-fowler]

I wonder if it's HTTP2. Async-http-client added support for that this year. I think you can create a async-http-client HTTPClient that forces HTTP1.1. Maybe you could try that.

[pballart]

I'm trying to proxy the requests through Charles to see what's exactly being sent. So far for the cRUL that works I'm seeing this:

It seems that HTTP2 should be working fine. I'm working on getting the Vapor request proxied to Charles.

[pballart]

Okay, got some juicy updates! Originally my code for the AWSClient was:

app.aws.client = AWSClient(httpClientProvider: .shared(app.http.client.shared))

and I don't remember why I was using this shared client, probably saw it in some documentation. 🤔

I changed that to enable the proxy:

app.aws.client = AWSClient(
        httpClientProvider: .shared(
            HTTPClient(
                eventLoopGroupProvider: .createNew,
                configuration: HTTPClient.Configuration(
                    proxy: HTTPClient.Configuration.Proxy.server(
                        host: "127.0.0.1",
                        port: 8888
                    )
                )
            )
        )
    )

and the request worked fine 🎉 , pointing to the issue being with the client.

The problem is that if I use

app.aws.client = AWSClient(httpClientProvider: .createNew)

or even

app.aws.client = AWSClient(
        httpClientProvider: .shared(
            HTTPClient(
                eventLoopGroupProvider: .createNew
            )
        )
    )

but without the proxy I'm still getting the same error...

Maybe this rings some bells for you, cause to me it doesn't make sense 🙃 Why proxying the request makes it work?

[adam-fowler]

This sounds like an AHC (async-http-client issue) issue, if the operation works fine when going through Charles, but doesn't when sent directly from AHC

[pballart]

I realized that probably the delete never worked but I just realized when I started using async/await. Could you reproduce the problem on your end or it's just me having this issue with the Delete operation?

[adam-fowler]

I don't have a Digital Ocean account. I have no issue deleting objects from AWS S3.

[adam-fowler]

Does it work if you use

app.aws.client = AWSClient(
        httpClientProvider: .shared(
            HTTPClient(
                eventLoopGroupProvider: .createNew,
                configuration: .init(httpVersion: .http1Only)
            )
        )
    )

[pballart]

Using .http1Only worked :) According to the docs HTTP2 should work as well and the curl also uses that but for some reason the AHC doesn't

[adam-fowler]

Can you add a bug to the AHC repo https://github.com/swift-server/async-http-client/issues?

[pballart]

Sure thing 👍 Thanks for your help!

[adam-fowler]

Closing as there is no more we can do at the Soto level here, given AHC removes content-size headers for DELETE operations