search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.07k stars 1.87k forks source link

Generate S3 presigned url with S3 Object Lambda Access Point not working #3678

Open sakurai-ryo opened 1 year ago

sakurai-ryo commented 1 year ago

Describe the bug

Hi Team. I am using S3 Object Lambdan Access Point to execute boto3 generate_presigned_url method. But when I access the generated URL, I get a MissingAuthenticationToken error.

If I use the @aws-sdk/s3-request-presigner module to generate it with JavaScript, I can access it via URL.

Expected Behavior

Being able to access S3 objects via a presigned URL.

Current Behavior

I got the following error in xml format

<Error>
  <Code>MissingAuthenticationToken</Code>
  <Message>Missing authentication token.</Message>
  <RequestId>a5b8fde1-2d48-4093-98e0-7883a955a1c6</RequestId>
  <HostId>{host-id}</HostId>
</Error>

Reproduction Steps

The minimal code to reproduce is below.

import boto3
import logging

boto3.set_stream_logger('', logging.DEBUG)

if __name__ == "__main__":
    sess = boto3.Session(profile_name="xxxxxx")
    s3_client = sess.client("s3")

    s3_object_lambda_access_point = "your object lambda access point arn"
    key = "your object key"

    url = s3_client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": s3_object_lambda_access_point, "Key": key},
        ExpiresIn=60,
    )
    print(url)

Possible Solution

No response

Additional Information/Context

Debug log for boto3 is below

% python presign.py
2023-04-21 12:31:57,908 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2023-04-21 12:31:57,909 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2023-04-21 12:31:57,909 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2023-04-21 12:31:57,910 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2023-04-21 12:31:57,910 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2023-04-21 12:31:57,910 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2023-04-21 12:31:57,911 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2023-04-21 12:31:57,912 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2023-04-21 12:31:57,912 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2023-04-21 12:31:57,912 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2023-04-21 12:31:57,912 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2023-04-21 12:31:57,912 botocore.session [DEBUG] Setting config variable for profile to 'xxxxxxxxx'
2023-04-21 12:31:57,915 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Skipping environment variable credential check because profile name was explicitly set.
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Looking for credentials via: sso
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2023-04-21 12:31:57,920 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2023-04-21 12:31:57,921 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/endpoints.json
2023-04-21 12:31:57,927 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2023-04-21 12:31:57,927 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x1048f39c0>
2023-04-21 12:31:57,937 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/service-2.json
2023-04-21 12:31:57,951 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2023-04-21 12:31:57,954 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/partitions.json
2023-04-21 12:31:57,955 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x104829b20>
2023-04-21 12:31:57,955 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x103739760>
2023-04-21 12:31:57,972 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x1048298a0>
2023-04-21 12:31:57,992 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2023-04-21 12:31:57,994 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/_retry.json
2023-04-21 12:31:58,003 botocore.client [DEBUG] Registering retry handlers for service: s3
2023-04-21 12:31:58,003 botocore.utils [DEBUG] Registering S3 region redirector handler
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x10491f7e0>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x109d08f50>>
2023-04-21 12:31:58,004 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': 'arn:aws:s3-object-lambda:ap-northeast-1:xxxxxxxxxxxxxx:accesspoint/test', 'Region': 'ap-northeast-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2023-04-21 12:31:58,004 botocore.regions [DEBUG] Endpoint provider result: https://xxxxxxxxxxx.s3-object-lambda.ap-northeast-1.amazonaws.com
2023-04-21 12:31:58,004 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2023-04-21 12:31:58,004 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'ap-northeast-1', 'signing_name': 's3-object-lambda', 'disableDoubleEncoding': True}
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x10491d580>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x10491d4e0>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x10491f600>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x109d08f50>>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x10491d300>
2023-04-21 12:31:58,005 botocore.hooks [DEBUG] Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x10982f210>>
2023-04-21 12:31:58,005 botocore.hooks [DEBUG] Event before-sign.s3.GetObject: calling handler <function remove_arn_from_signing_path at 0x10491f740>
2023-04-21 12:31:58,005 botocore.auth [DEBUG] Calculating signature using hmacv1 auth.
2023-04-21 12:31:58,005 botocore.auth [DEBUG] HTTP request method: GET
2023-04-21 12:31:58,005 botocore.auth [DEBUG] StringToSign:

SDK version used

1.26.117

Environment details (OS name and version, etc.)

MacOS Ventura

tim-finnigan commented 1 year ago

Hi @sakurai-ryo thanks for reaching out. Have you tried using SigV4 as suggested in this re:Post answer? For example: s3 = boto3.client('s3', config=Config(signature_version='s3v4'))

sakurai-ryo commented 1 year ago

@tim-finnigan It worked correctly, thank you! I will send a PR to add this to the documentation. https://github.com/boto/boto3/blob/develop/docs/source/guide/s3-presigned-urls.rst