TransferUtility does not support downloading objects in Requester Pays buckets.

krishsingh-git commented 6 months ago

Describe the bug

await s3Client.DownloadToFilePathAsync( s3Location.BucketName.Value, s3Location.Key.Value, downloadPath.Path.Value, additionalProperties);

Expected Behavior

TransferUtility does not support downloading objects in Requester Pays buckets. allow download ,, remove access denied error

Current Behavior

TransferUtility does not support downloading objects in Requester Pays buckets.

await s3Client.DownloadToFilePathAsync( s3Location.BucketName.Value, s3Location.Key.Value, downloadPath.Path.Value, additionalProperties);

Reproduction Steps

await s3Client.DownloadToFilePathAsync( s3Location.BucketName.Value, s3Location.Key.Value, downloadPath.Path.Value, additionalProperties);

Possible Solution

add requestpayer in transfer utility DownloadToFilePathAsync

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.9.17

Targeted .NET Platform

.net core 3.1

Operating System and version

window

krishsingh-git commented 6 months ago

its high priority ?can anyone look into issue

bhoradc commented 6 months ago

Hi @krishsingh-git,

Thank you for reporting the issue. If you came across the Access denied error, when downloading object from a Requester Pays bucket using TransferUtility, can you kindly provide below additional information?

Self-contained code sample that reproduces the Access Denied error.
Does the Source and destination buckets reside in cross-accounts?

Regards, Chaitanya

krishsingh-git commented 6 months ago

Hi,

We are using below method to download file

await s3Client.DownloadToFilePathAsync( s3Location.BucketName.Value, s3Location.Key.Value, downloadPath.Path.Value, additionalProperties);

Does the Source and destination resides in cross-accounts?
Source: is S3 buclet destination is netwok path.

From: bhoradc @.> Sent: Thursday, April 25, 2024 7:52 PM To: aws/aws-sdk-net @.> Cc: Krishan Singh @.>; Mention @.> Subject: Re: [aws/aws-sdk-net] TransferUtility does not support downloading objects in Requester Pays buckets. (Issue #3301)

CAUTION: This email is from an external source. Please don't open any unknown links or attachments.

Hi @krishsingh-githttps://github.com/krishsingh-git,

Thank you for reporting the issue. If you came across the Access denied error, when downloading object from a Requester Pays bucket using TransferUtility, can you kindly provide below additional information?

Self-contained code sample that reproduces the Access Denied error.
Does the Source and destination resides in cross-accounts?

Regards, Chaitanya

- Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-sdk-net/issues/3301#issuecomment-2077329295, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BADY3SMY5FR25GOXVZD2BF3Y7EGRNAVCNFSM6AAAAABGYY6JWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZXGMZDSMRZGU. You are receiving this because you were mentioned.Message ID: @.**@.>> This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service https://www.amdocs.com/about/email-terms-of-service

krishsingh-git commented 6 months ago

please make it P1,, in our organisation ,, its burning issue

krishsingh-git commented 6 months ago

please arrange a call i will explain

bhoradc commented 6 months ago

Hi @krishsingh-git,

I am able to reproduce the Access Denied AmazonS3Exception when downloading objects using the Transfer utility with Requester Pays enabled.

Below are the steps and the code sample I used with the latest .NET SDK/Package version for reproduction.

Enabled Requester pays option from the console for the bucket in account A.
Configured the necessary bucket policies for s3:GetObject for the IAM role from account B.
Using below snippet, I am getting the S3 access exception when downloading the object (account A) from account B.

using Amazon.S3;
using Amazon.S3.Transfer;
using System;
using System.Threading.Tasks;

namespace ReqPaysIssue_3301
{
    class Program
    {
        static async Task Main(string[] args)
        {
            try
            {
                IAmazonS3 s3Client = new AmazonS3Client();
                var transferUtil = new TransferUtility(s3Client);
                await transferUtil.DownloadAsync(new TransferUtilityDownloadRequest
                {
                    BucketName = <<bucket_name>>,
                    Key = <<key>>,
                    FilePath = <<file_path>>,
                });
            }
            catch (Exception ex)
            {
                await Console.Out.WriteLineAsync(ex.Message);
                throw;
            } 
        }     
    }    
}

Reproduced output:

Access Denied Unhandled exception. Amazon.S3.AmazonS3Exception: Access Denied ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.

Kindly let me know if above reproduction is unclear or if you have any other observations.

I will review this issue with the SDK team and keep you posted on the updates.

Regards, Chaitanya

krishsingh-git commented 6 months ago

Yes correct ,, Getting same issue,, Please check request payer issue across SDK, because same issue we face in multipart copy.it was fixed,, but not for download.. Its very urgent ,, very high pressure ☹

From: bhoradc @.> Sent: Saturday, April 27, 2024 10:09 AM To: aws/aws-sdk-net @.> Cc: Krishan Singh @.>; Mention @.> Subject: Re: [aws/aws-sdk-net] TransferUtility does not support downloading objects in Requester Pays buckets. (Issue #3301)

CAUTION: This email is from an external source. Please don’t open any unknown links or attachments.

Hi @krishsingh-githttps://github.com/krishsingh-git,

I am able to reproduce the Access Denied AmazonS3Exception when downloading objects using the Transfer utility with Requester Pays enabled.

Below are the steps and the code sample I used with the latest .NET SDK/Package version for reproduction.

Enabled Requester pays option from the console for the bucket in account A.
Configured the necessary bucket policies for s3:GetObject for the IAM role from account B.
Using below snippet, I am getting the S3 access exception when downloading the object (account A) from account B.

using Amazon.S3;

using Amazon.S3.Transfer;

using System;

using System.Threading.Tasks;

namespace ReqPaysIssue_3301

{

class Program

{

    static async Task Main(string[] args)

    {

        try

        {

            IAmazonS3 s3Client = new AmazonS3Client();

            var transferUtil = new TransferUtility(s3Client);

            await transferUtil.DownloadAsync(new TransferUtilityDownloadRequest

            {

                BucketName = <<bucket_name>>,

                Key = <<key>>,

                FilePath = <<file_path>>,

            });

        }

        catch (Exception ex)

        {

            await Console.Out.WriteLineAsync(ex.Message);

            throw;

        }

    }

}

}

Reproduced output:

Access Denied Unhandled exception. Amazon.S3.AmazonS3Exception: Access Denied ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.

Kindly let me know if above reproduction is unclear or if you have any other observations.

I will review this issue with the SDK team and keep you posted on the updates.

Regards, Chaitanya

— Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-sdk-net/issues/3301#issuecomment-2080358296, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BADY3SMYUJS7A5SA7L4IOGLY7MTYDAVCNFSM6AAAAABGYY6JWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBQGM2TQMRZGY. You are receiving this because you were mentioned.Message ID: @.**@.>> This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service https://www.amdocs.com/about/email-terms-of-service

ashishdhingra commented 6 months ago

@krishsingh-git RequestPayer property should be available in AWSSDK.S3 version 3.7.307.27 (refer Chengelog)

github-actions[bot] commented 6 months ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

krishsingh-git commented 6 months ago

Thank you so much for help.

I would suggest please reverify across AWSSDKS3 .net, if anywhere else required to add/enable "RequestPayer".

From: Ashish Dhingra @.> Sent: Friday, May 3, 2024 3:41 AM To: aws/aws-sdk-net @.> Cc: Krishan Singh @.>; Mention @.> Subject: Re: [aws/aws-sdk-net] TransferUtility does not support downloading objects in Requester Pays buckets. (Issue #3301)

CAUTION: This email is from an external source. Please don't open any unknown links or attachments.

Closed #3301https://github.com/aws/aws-sdk-net/issues/3301 as completed.

- Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-sdk-net/issues/3301#event-12689941714, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BADY3SOEJ62WUMHRX7L6UYDZAK2YBAVCNFSM6AAAAABGYY6JWWVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSGY4DSOJUGE3TCNA. You are receiving this because you were mentioned.Message ID: @.**@.>> This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service https://www.amdocs.com/about/email-terms-of-service

aws / aws-sdk-net