AWS S3 SDK: Error when calling GetPreSignedURL for certain regions

[jerome-zenfolio]

Describe the bug

Hello!

I am using the S3 SDK to for non-Amazon endpoints specifically Wasabi. When generating presigned URL for certain regions, it fails with the message: The maximum expiry period for a presigned url using AWS4 signing is 604800 seconds

The regions that fails are: https://s3.us-east-2.wasabisys.com https://s3.us-central-1.wasabisys.com

But it works okay for: https://s3.us-west-1.wasabisys.com https://s3.us-east-1.wasabisys.com

The below xUnit test should work as is with no changes.

Thank you kindly!

Expected Behavior

Expect the presigned URL to be generated correctly.

Current Behavior

Message: 

    System.ArgumentException : The maximum expiry period for a presigned url using AWS4 signing is 604800 seconds

  Stack Trace: 
    AmazonS3Client.Marshall(IClientConfig config, GetPreSignedUrlRequest getPreSignedUrlRequest, String accessKey, String token, SignatureVersion signatureVersion)
    AmazonS3Client.GetPreSignedURLInternal(GetPreSignedUrlRequest request, Boolean useSigV2Fallback)
    AmazonS3Client.GetPreSignedURL(GetPreSignedUrlRequest request)
    S3VolumeOperationShould.PresignedShould(String serviceURL) line 267

Reproduction Steps

        [Theory]
        [InlineData("https://s3.us-west-1.wasabisys.com")] // pass
        [InlineData("https://s3.us-east-1.wasabisys.com")] // pass
        [InlineData("https://s3.us-east-2.wasabisys.com")] // error
        [InlineData("https://s3.us-central-1.wasabisys.com")] // error
        public void PresignedShould(string serviceURL)
        {
            var client = new AmazonS3Client(
                "Settings.AuthKeyId",
                "Settings.AuthKey",
                new AmazonS3Config { ServiceURL = serviceURL });

            client.GetPreSignedURL(new GetPreSignedUrlRequest
            {
                BucketName = "bucket",
                Expires = DateTime.UtcNow.AddDays(10),
                Verb = HttpVerb.GET
            });
        }

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

3.7.9.54, 3.7.108.

Targeted .NET Platform

.NET 6

Operating System and version

Windows & Ubuntu

[ashishdhingra]

@jerome-zenfolio Good morning. Thanks for reporting the issue. With new version of AWS SDK, the region and endpoint resolution has changed. Also, we do not support usage of AWS SDKs with 3rd party services, although it might work in certain scenarios (which is good).

To troubleshoot your scenario, you may develop a console application, turn on verbose logging using below code and investigate any useful information from logs:

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

Thanks, Ashish

[jerome-zenfolio]

Indeed it is good thing: we've used many 3rd party S3 vendors using this SDK without any issues and thank you for maintaining such a versatile SDK!

As requested, I have added the above logging and attaching the output below for one of the failed cases:

    Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy]
    Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy].
    Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy].
    Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy].
    Unable to find exact matched region in endpoint s3.us-central-1.wasabisys.com
    us-central-1 fuzzy region found in endpoint s3.us-central-1.wasabisys.com

Looking at the SDK implementation, it looks like for certain regions, it automatically downgrades to signature v2 and generates the URL successfully.

[ashishdhingra]

Indeed it is good thing: we've used many 3rd party S3 vendors using this SDK without any issues and thank you for maintaining such a versatile SDK!

As requested, I have added the above logging and attaching the output below for one of the failed cases:

    Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy]
    Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy].
    Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy].
    Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy].
    Unable to find exact matched region in endpoint s3.us-central-1.wasabisys.com
    us-central-1 fuzzy region found in endpoint s3.us-central-1.wasabisys.com

Looking at the SDK implementation, it looks like for certain regions, it automatically downgrades to signature v2 and generates the URL successfully.

@jerome-zenfolio I'm unsure what's happening in your case, however, SDK:

• Would use SigV2 is the region is resolved as us-east-1 as per line.
• If the expiration time is longer than what SigV4 will allow then it would automatically use SigV2 instead as per line.

Again as mentioned earlier, we do not support usage of AWS SDKs with 3rd party services. So based on above inputs, please try to troubleshoot your use case or open a case with wasabi support.

Thanks, Ashish

[jerome-zenfolio]

Thanks @ashishdhingra, for the response! As you can tell from the unit tests that I have given earlier, the automatic downgrade did not happen and hence the ticket is created. Also, since the presigned request is not sent to the server, I am not quite certain how Wasabi support can help me here.

Here is the boto3 SDK that correctly generates the presigned URL without any error for the very same endpoint that failed in .NET SDK:

import boto3
import datetime

def main():
    expiration_duration = 2592000

    s3_client = boto3.client(
        's3', 
        aws_access_key_id='Settings.AuthKeyId',
        aws_secret_access_key='Settings.AuthKey',
        endpoint_url='https://s3.us-east-2.wasabisys.com')

    expiration_time = datetime.datetime.now() + datetime.timedelta(seconds=expiration_duration)

    presigned_url = s3_client.generate_presigned_url(
        'get_object',
        Params={
            'Bucket': 'bucket',
            'Key': 'key'
        },
        ExpiresIn=expiration_duration
    )

    print("Presigned URL:", presigned_url)

if __name__ == "__main__":
    main()

.NET SDK on the other hand, fails for the same endpoint.

Thanks, Jerome

[ashishdhingra]

@jerome-zenfolio Per MaxAWS4PreSignedUrlExpiry constant, 7 days (604800 seconds) is the maximum expiration for SigV4 presigned URL(s). If the s3SignatureVersionOverride for your endpoint is somehow resolved to 2, then it would use signature version 2 (refer links in https://github.com/aws/aws-sdk-net/issues/3003#issuecomment-1648249519).

As mentioned earlier, we cannot troubleshoot issues with SDK when used with non-AWS services. I'm unsure how the endpoint rules for wasabi service are resolved and could not comment on how it works with Boto3.

Thanks, Ashish

[chris-zenfolio]

Hi Ashish,

The SDKs we use for the S3 standard come from AWS. We primarily use the .NET SDK, but this is clearly shown to have a bug. The Boto3 SDK (for Python) is an AWS product with information about it at https://aws.amazon.com/sdk-for-python/.

We opened a case with Wasabi last week and they referred us to AWS since Boto3 works. We are not asking you to troubleshoot Wasabi's ecosystem, rather fix a bug in the .NET SDK that has inconsistencies between it's "sister SDK" for Python (Boto3).

Since generating a pre-signed URL does not actually contact any server, this has to be a bug in the SDK. @jerome-zenfolio already demonstrated how it does not fall-back to SigV2 when appropriate.

Thanks, Chris

[ashishdhingra]

Hi Ashish,

The SDKs we use for the S3 standard come from AWS. We primarily use the .NET SDK, but this is clearly shown to have a bug. The Boto3 SDK (for Python) is an AWS product with information about it at https://aws.amazon.com/sdk-for-python/.

We opened a case with Wasabi last week and they referred us to AWS since Boto3 works. We are not asking you to troubleshoot Wasabi's ecosystem, rather fix a bug in the .NET SDK that has inconsistencies between it's "sister SDK" for Python (Boto3).

Since generating a pre-signed URL does not actually contact any server, this has to be a bug in the SDK. @jerome-zenfolio already demonstrated how it does not fall-back to SigV2 when appropriate.

Thanks, Chris

@chris-zenfolio I will discuss this with the team and post any update here. However, since the SDK works perfectly fine with the AWS system and recent Endpoints 2.0 implementation changes the way endpoints are resolved, cannot commit if this issue would be prioritized by the team.

Thanks, Ashish

[normj]

@jerome-zenfolio Boto working for us-east-2 and us-central-1 is by accident not design. I would bet if you went across the 12 AWS SDKs you would get a collection of behavior for specifying a non-AWS endpoint and then using an expiration greater then the SigV4 max. In fact most of the newer SDKs only support SigV4 with no SigV2 fallback.

The .NET SDK, which is assuming an AWS endpoint, looks at the service URL to determine the AWS region the request is being sent to. The parse logic for the host name is assuming the service url is <service>.<region>.<host-suffix> Which the Wasabi host name follow so for https://s3.us-east-2.wasabisys.com/ the .NET SDK determine the request is being sent to the us-east-2 AWS region which is a SigV4 only S3 region. For https://s3.us-central-1.wasabisys.com/ there is no AWS region called us-central-1 so the SDK is assuming you are attempting to use a new region and all new regions are SigV4 only regions so it defaults to that.

SigV4 has the restriction that signature can not last any longer then 7 days so in your case when the SDK determines it needs to use SigV4 for the signature it fails since you are using 10 days. Can you use 7 instead of 10 days for your expiration?

The SDK's logic for determine regions and signing is very complicated with a lot of built up legacy decisions with how AWS regions have changed over the years and how SDKs are supposed to resolve endpoints and signing regions. I'm very leery of adding more complications to the code for a scenario that shouldn't be valid since it violates SigV4 requirements.

[chris-zenfolio]

Thanks for the replies, @normj and @ashishdhingra .

For the signature validity duration, we actually need 60 days. Our use case is to provide customers with a pre-signed URL that is valid for 60 days so they can download the zip file of their photos without us having an open bucket. We have found that 7 days is far too short. Thus, SigV4 is not feasible for us to use for this purpose.

[normj]

I have no idea if this will work for Wasabi but worth a try. When you construct the the AmazonS3Config object try always setting the AuthenticationRegion to us-east-1 even when the ServiceUrl is to some other region. That should trick the SDK into using SigV2 which doesn't include the region as part of the signing. Not sure what Wasabi will do in this case but you can see if it unblocks you.

var client = new AmazonS3Client(
    "Settings.AuthKeyId",
    "Settings.AuthKey",
    new AmazonS3Config { ServiceURL = serviceURL, AuthenticationRegion = "us-east-1" });

Keep in mind this unsupported behavior but maybe it will unblock you.

[jerome-zenfolio]

@normj, thank you for the detailed explanation and for the workaround. I was able to generate presigned url with expiration longer than 7 days using the AuthenticationRegion override. We understand this is unsupported and the behavior can change over time but seems like a good workaround for the immediate needs. Thanks!

[chris-zenfolio]

@normj Would Amazon be amenable if we created a PR that enables the 3rd party services who have implemented the S3 API. Effectively we would just wrap a check for amazonaws.com around the code that looks up and enforces the AWS region. Below is some pseudo code that illustrates the concept only.

if (domain == "amazonaws.com")
    check whether SigV2 is allowed in this AWS region and set appropriate flags/act accordingly
else
    continue processing with no region restrictions

[normj]

@chris-zenfolio There is actually a push to end support for SigV2 which is why new SDKs don't have support for SigV2. I'm not announcing the end of SigV2 support but if/when we do a V4 of the SDK it would likely not have SigV2 support anymore. I think what this mean is products that are emulating S3 really need emulate SigV4 otherwise many SDKs won't work.

So I'm leery for changing this code for this use case. It is more complicated then you suggest above because there are many S3 endpoints across the world that don't have an amazonaws.com suffix and then you throw in things like VPC endpoints which creates custom service endpoints.

Sorry not trying to be a jerk. I want to protect some complicated code from getting even more complicated.

[ashishdhingra]

Closing this issue since no action needs to be taken from your side based on last comment.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.