Deleting S3 object fails on .NET for iOS.

joduss commented 9 months ago

Describe the bug

We are using the AWS SDK for .Net in an iOS application built using .NET 8.

We are observing a possible bug when trying to delete an object in a bucket:

In the iOS application, we constantly get an exception (AmazonS3Exception: A header you provided implies functionality that is not implemented) while the same code works fine in a .NET 8 console application. Before migrating the app from Xamarin.iOS to .NET 8, it used to work fine as well.

Expected Behavior

The object should be successfully deleted.

Current Behavior

The following exception is thrown:

AmazonS3Exception: A header you provided implies functionality that is not implemented
                     ▷ Trace:
                       ▷ at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
                       ▷ at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
                       ▷ at Amazon.Runtime.Internal.ExceptionHandler`1.<HandleAsync>d__6[[Amazon.Runtime.Internal.HttpErrorResponseException, AWSSDK.Core, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
                       ▷ at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.Signer.<InvokeAsync>d__1`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                     ► HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
                       ▷ Trace:
                         ▷ at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
                         ▷ at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
                         ▷ at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1[[System.Net.Http.HttpContent, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                         ▷ at Amazon.Runtime.Internal.RedirectHandler.<InvokeAsync>d__1`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                         ▷ at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                         ▷ at Amazon.S3.Internal.AmazonS3ResponseHandler.<InvokeAsync>d__1`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                         ▷ at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()

Reproduction Steps

Use the following code in an application, replacing uppercased values.

var bucket = "BUCKET_NAME";
var credentials = new BasicAWSCredentials("ACCESS_KEY", "SECRET_KEY");

using var s3Client = new AmazonS3Client(credentials, BUCKET_REGION);

var deleteObjectRequest = new DeleteObjectRequest {
    BucketName = bucket,
    Key = "OBJECT_ID",
};

try {
    await s3Client.DeleteObjectAsync(deleteObjectRequest);
} catch (Exception e) {
    Console.WriteLine(e.StackTrace);
}

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.305.5

Targeted .NET Platform

.NET8-iOS

Operating System and version

iOS 17

ashishdhingra commented 9 months ago

@joduss Good morning. The mentioned error is returned by the S3 service, refer S3: List of error codes. Perhaps, somehow on iOS, there might be erroneous header being sent to S3 service which is causing the error.

Is it Xamarin iOS application targeting .NET 8?
Are you using the same object key in iOS application as in working Console application?
Does object key contain any special characters (like double quote)?

Could you please enable verbose logging using the code below and share the detailed logs which might capture some information sent down to the wire:

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.Console;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

Thanks, Ashish

joduss commented 9 months ago

Now regarding your questions:

Is it Xamarin iOS application targeting .NET 8?
- Yes it is.
Are you using the same object key in iOS application as in working Console application?
- Yes

I will share the logs next week.

Workaround (Does not work)

I thought I found a workaround. Somehow it did work for a while but does not anymore: _Initial answer: I did found in the meantime a workaround by instantiating the client in a different way and specifying the `DefaultConfigurationMode`_ ```csharp new AmazonS3Client( Credential.AccessKey, Credential.SecretKey, new AmazonS3Config { RegionEndpoint = RegionEndpoint.GetBySystemName(regionResponse.Location), DefaultConfigurationMode = DefaultConfigurationMode.Standard // This fixes things. } ); ```

joduss commented 9 months ago

Here are the logs:

verbose.log

ashishdhingra commented 9 months ago

@joduss Good morning. Thanks for the response and confirming that the workaround initially shared by you in https://github.com/aws/aws-sdk-net/issues/3140#issuecomment-1878921261 when setting DefaultConfigurationMode to DefaultConfigurationMode.Standard, since we also thought that the error and setting configuration mode explicitly were unrelated.

In the logs, I see the below error:

...
2024-01-08 09:48:07.676911+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
2024-01-08 09:48:07.685057+0100 My-App.iOS[57738:6780336] AmazonS3Client 29|2024-01-08T09:48:07.684Z|ERROR|Received error response: [<?xml version="1.0" encoding="UTF-8"?>
2024-01-08 09:48:07.685280+0100 My-App.iOS[57738:6780336] <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>If-Modified-Since</Header><RequestId>BH3YCB93BZMKV75S</RequestId><HostId>OfT15rgmuSK9S8dt+HlGuRFrki/e9NOmX5sAZOvGf1olr5SyKPv9zmOPGI4DaoGMjRUaaCDnd0cBo3IyyiIS+A==</HostId></Error>] --> Amazon.S3.AmazonS3Exception: A header you provided implies functionality that is not implemented
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
2024-01-08 09:48:07.685448+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
...
...
   --- End of inner exception stack trace ---
2024-01-08 09:48:07.696940+0100 My-App.iOS[57738:6780336] AmazonS3Client 30|2024-01-08T09:48:07.695Z|ERROR|AmazonS3Exception making request DeleteObjectRequest to https://dev-bucket.s3.eu-central-1.amazonaws.com/. Attempt 1. --> Amazon.S3.AmazonS3Exception: A header you provided implies functionality that is not implemented
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
2024-01-08 09:48:07.697156+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
2024-01-08 09:48:07.697360+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1[[System.Net.Http.HttpContent, System.Net.Http, Version=8.0.0.0, Culture=neutral, 
...
...
2024-01-08 09:48:07.699023+0100 My-App.iOS[57738:6780336]    at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
2024-01-08 09:48:07.699157+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
...

It mentions error <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>If-Modified-Since</Header><RequestId>BH3YCB93BZMKV75S</RequestId><HostId>OfT15rgmuSK9S8dt+HlGuRFrki/e9NOmX5sAZOvGf1olr5SyKPv9zmOPGI4DaoGMjRUaaCDnd0cBo3IyyiIS+A==</HostId></Error>

I'm unsure if If-Modified-Since is causing the issue and how it is being populated for your scenario.

Could you please share the following:

Are you using the same bucket for Console and Xamarin iOS app scenario?
Is your bucket S3 Express bucket? Does it has directory style bucket? (refer Directory buckets section in What is S3 Express One Zone?)
What does your object key looks like? (in other words, does it contain any special characters?)

Similar issue https://github.com/aws/aws-sdk-js/issues/2787, but with no concrete information about the root cause or resolution.

Thanks, Ashish

joduss commented 9 months ago

Are you using the same bucket for Console and Xamarin iOS app scenario?
- I am using the same bucket and trying to delete the same object.
- The object was uploaded sometimes using the AWS SDK for .NET, sometimes using Cyberduck.
Is your bucket S3 Express bucket? Does it has directory style bucket? (refer Directory buckets section in What is S3 Express One Zone?)
- No
What does your object key looks like? (in other words, does it contain any special characters?)
- The object is actually a zip file with the extension being ".appname-instruction". The id of that object is "devices/gQFYdfd8vEil8xyDYpV7GQ/iHJ4KEAlmkWktafe84V2WA.appname-instruction". I have replace the actual application name by "appname". The real name is using only chars in [a-z]. So no special characters in the identifier.

muhammad-othman commented 9 months ago

Good day @joduss, Based on the error that you are getting, I suspect that something is modifying the request headers before it is sent to S3, maybe this related to iOS or something else. Can you intercept the request and provide it including the headers?

joduss commented 9 months ago

This is the request I captured from the .Net iOS app:

DELETE https://bucket-name.s3.eu-central-1.amazonaws.com/devices/gQFYdfd8vEil8xyDYpV7GQ/iHJ4KEAlmkWktafe84V2WA.myapp-instruction HTTP/1.1
Host: bucket-name.s3.eu-central-1.amazonaws.com
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/20240105/eu-central-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d46489c822f6079ca995f8a22d49c2a0b7d43dab09979ea735540bf547184aa0
If-Modified-Since: Fri, 05 Jan 2024 13:21:57 GMT
X-Amz-Date: 20240105T153804Z
Accept-Encoding: gzip, deflate, br
amz-sdk-invocation-id: 7acaa70f-ec17-402b-942e-8e94f9f7f75c
amz-sdk-request: attempt=1; max=5
Accept-Language: en-US,en;q=0.9
If-None-Match: "13f6daf619d5ae220656c30c7d63c21c"
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
User-Agent: aws-sdk-dotnet-coreclr/3.7.305.5 ua/2.0 os/other md/Darwin 22.6.0 Darwin Kernel Version 22.6.0- Wed Oct 4 21-25-26 PDT 2023- root-xnu-8796.141.3.701.17~4/RELEASE_X86_64#17.0.1 md/ARCH#X64 lang/.NET_Core#8.0.0 md/aws-sdk-dotnet-core#3.7.301.0 api/S3#3.7.305.5 ft/aot cfg/retry-mode#legacy md/ClientAsync
Content-Length: 0
Connection: keep-alive

Now for comparison, the request made the console app:

DELETE https://bucket-name.s3.eu-central-1.amazonaws.com/devices/gQFYdfd8vEil8xyDYpV7GQ/iHJ4KEAlmkWktafe84V2WA.myapp-instruction HTTP/1.1
User-Agent: aws-sdk-dotnet-coreclr/3.7.305.5 ua/2.0 os/macos#13.6.1 md/ARCH#X64 lang/.NET_Core#8.0.0 md/aws-sdk-dotnet-core#3.7.301.0 api/S3#3.7.305.5 cfg/retry-mode#legacy md/ClientAsync
amz-sdk-invocation-id: d1ea090b-25a3-46de-ab7d-39c9b87aa6f9
amz-sdk-request: attempt=1; max=5
Host: bucket-name.s3.eu-central-1.amazonaws.com
X-Amz-Date: 20240105T153814Z
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/20240105/eu-central-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f01e8c1e7ccb49f8948dfdea33120aaa3438cf8c9f7d3ddd60a1fd00681197e5

joduss commented 9 months ago

There is one additional observation I made:

If the file is very small, the delete requests fails as explained above. If the file is larger, it succeeds. The header differs depending if the request is to delete a large file or a small file.

Header of request to delete a file of size 1085 bytes:

DELETE https://backup-name.s3.eu-central-1.amazonaws.com/devices/gQFYdfd8vEil8xyDYpV7GQ/a.myapp-instruction HTTP/1.1
Host: backup-name.s3.eu-central-1.amazonaws.com
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_LEY/20240111/eu-central-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=a2cda8ee204b09aaba7fa6ec890f3c4474aaeb77b61418e1c7975d03ce9b26ba
If-Modified-Since: Thu, 11 Jan 2024 10:42:05 GMT
X-Amz-Date: 20240111T104209Z
Accept-Encoding: gzip, deflate, br
amz-sdk-invocation-id: 188175ed-6834-4c9a-b103-159915b80472
amz-sdk-request: attempt=1; max=5
Accept-Language: en-US,en;q=0.9
If-None-Match: "9aeccf7335aa2afe20282060ae026371"
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
User-Agent: aws-sdk-dotnet-coreclr/3.7.304.0 aws-sdk-dotnet-core/3.7.300.11 .NET_Core/8.0.1 OS/Darwin_22.6.0_Darwin_Kernel_Version_22.6.0:_Wed_Oct__4_21:25:26_PDT_2023;_root:xnu-8796.141.3.701.17~4/RELEASE_X86_64 ft/aot ClientAsync
Content-Length: 0
Connection: keep-alive

Header of request to delete a file of size 903'440 bytes:

DELETE https://backup-name.s3.eu-central-1.amazonaws.com/devices/gQFYdfd8vEil8xyDYpV7GQ/b.myapp-instruction HTTP/1.1
Host: backup-name.s3.eu-central-1.amazonaws.com
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/20240111/eu-central-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9bfe86c01072568dfda3b2154a5af084af0b4ea77c53aec9820bcfe4bf42acd0
X-Amz-Date: 20240111T105236Z
Accept-Encoding: gzip, deflate, br
amz-sdk-invocation-id: 48057b06-5d22-4157-b4a4-adbee04b0b1c
amz-sdk-request: attempt=1; max=5
Accept-Language: en-US,en;q=0.9
Content-Length: 0
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
User-Agent: aws-sdk-dotnet-coreclr/3.7.304.0 aws-sdk-dotnet-core/3.7.300.11 .NET_Core/8.0.1 OS/Darwin_22.6.0_Darwin_Kernel_Version_22.6.0:_Wed_Oct__4_21:25:26_PDT_2023;_root:xnu-8796.141.3.701.17~4/RELEASE_X86_64 ft/aot ClientAsync
Connection: keep-alive

muhammad-othman commented 9 months ago

Thank you for sharing the requests, I couldn't find anything that might cause this issue from the SDK side. I've reached out to S3 with your request to figure out the root cause, will let you know when then respond.

ashishdhingra commented 7 months ago

V1197046386

aws / aws-sdk-net