search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.81k stars 1.84k forks source link

using boto3.client(xxxxx).get_object donw data,It speed have very difference. #4143

Open GBY-1 opened 1 month ago

GBY-1 commented 1 month ago

Describe the bug

I use get object donw sentinel data from 'Bucket=sentinel-2-l1c' and 'Bucket=landsat-c2l1'. sentinel-2-l1c down is very slow,but i use aws cli dont this issue.please help me to address this issue

Expected Behavior

promote down speed

Current Behavior

down speed very slow

Reproduction Steps

file_size = int(s3.head_object(Bucket=bucket_name, Key=key, RequestPayer='requester')['ContentLength']) - 1 with open(filepath, 'wb') as f: while True: try: if down_trycount > 2: break

使用range参数获取要下载的字节范围

        end_byte = start_byte + chunk_size - 1
        if end_byte > file_size:
            end_byte = file_size
        range_str = f"bytes={start_byte}-{end_byte}"
        print(range_str)

        # 下载块并将其追加到本地文件
        resp = s3.get_object(Bucket=bucket_name, Key=key, RequestPayer='requester', Range=range_str)
        to_write = resp['Body'].read()
        print('正在写入------',range_str)
        f.seek(start_byte)
        f.write(to_write)
        # 更新起始位置以准备下一块的下载
        start_byte += chunk_size
        # 每一个文件的小分片，下载失败都会重新下载三次（下载成功则次数初始化为0）
        down_trycount = 0
        # 下载成功
        if start_byte > file_size:
            down_item_success = 'success'
            break
    except Exception as e:
        down_trycount = down_trycount + 1
        print(key, '========下载失败', e)

f.close()

Possible Solution

No response

Additional Information/Context

No response

SDK version used

boto3

Environment details (OS name and version, etc.)

boto3 python3

tim-finnigan commented 3 weeks ago

Thanks for reaching out. Have you reviewed the performance guidelines for S3 here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html ? Please let us know what you have tried thus far.

Both the AWS CLI and Boto3 use Botocore, so I would expect to see similar download speeds. Are you using the latest version of Boto3? And what difference in download speed are you seeing?

Please provide a complete code snippet as well as debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('') to your script for us to investigate this further.

GBY-1 commented 2 weeks ago

environment: Name: boto3 Version: 1.34.113

my python code are follow as:(aws account is privacy,I dont provide.For example:USERNAME,PASSWORD ) `import boto3 from datetime import datetime from io import BytesIO from botocore.config import Config import logging

bucket_name = 'sentinel-s2-l1c' s3_object_key = 'tiles/49/T/EE/2023/7/4/0/B01.jp2' profix = 'tiles/49/T/EE/2023/7/4/0/'

start_byte = 0 per_byte = 1024 1024 2 filepath = 'F:/gbytemp/test.tif'

boto3.set_stream_logger('') config = Config(connect_timeout=10, read_timeout=30, region_name='eu-central-1') s3 = boto3.resource('s3',aws_access_key_id=USERNAME,aws_secret_access_key=PASSWORD,config=config) bucket = s3.Bucket(bucket_name) object = s3.Object(bucket_name,s3_object_key) starttime = datetime.now() print(starttime) response = object.get(RequestPayer='requester') content = response['Body'] with open(filepath, 'wb') as f: while True: towrite = content.read(per_byte) if not towrite: break f.seek(start_byte) f.write(towrite) start_byte = start_byte + per_byte print(start_byte) `

loggings show: `2024-06-13 14:32:43,280 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x000002DB22C86670> 2024-06-13 14:32:43,280 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x000002DB22C86670> 2024-06-13 14:32:43,282 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x000002DB22BF9700> 2024-06-13 14:32:43,282 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x000002DB22BF9700> 2024-06-13 14:32:43,283 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.._handler at 0x000002DB22649040> 2024-06-13 14:32:43,283 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.._handler at 0x000002DB22649040> 2024-06-13 14:32:43,284 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x000002DB22BF94C0> 2024-06-13 14:32:43,284 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x000002DB22BF94C0> 2024-06-13 14:32:43,285 botocore.configprovider [DEBUG] Looking for endpoint for s3 via: environment_service 2024-06-13 14:32:43,285 botocore.configprovider [DEBUG] Looking for endpoint for s3 via: environment_service 2024-06-13 14:32:43,286 botocore.configprovider [DEBUG] Looking for endpoint for s3 via: environment_global 2024-06-13 14:32:43,286 botocore.configprovider [DEBUG] Looking for endpoint for s3 via: environment_global 2024-06-13 14:32:43,288 botocore.configprovider [DEBUG] Looking for endpoint for s3 via: config_service 2024-06-13 14:32:43,288 botocore.configprovider [DEBUG] Looking for endpoint for s3 via: config_service 2024-06-13 14:32:43,289 botocore.configprovider [DEBUG] Looking for endpoint for s3 via: config_global 2024-06-13 14:32:43,289 botocore.configprovider [DEBUG] Looking for endpoint for s3 via: config_global 2024-06-13 14:32:43,290 botocore.configprovider [DEBUG] No configured endpoint found. 2024-06-13 14:32:43,290 botocore.configprovider [DEBUG] No configured endpoint found. 2024-06-13 14:32:43,347 botocore.endpoint [DEBUG] Setting s3 timeout as (10, 30) 2024-06-13 14:32:43,347 botocore.endpoint [DEBUG] Setting s3 timeout as (10, 30) 2024-06-13 14:32:43,350 botocore.client [DEBUG] Registering retry handlers for service: s3 2024-06-13 14:32:43,350 botocore.client [DEBUG] Registering retry handlers for service: s3 2024-06-13 14:32:43,351 botocore.utils [DEBUG] Registering S3 region redirector handler 2024-06-13 14:32:43,351 botocore.utils [DEBUG] Registering S3 region redirector handler 2024-06-13 14:32:43,352 botocore.utils [DEBUG] Registering S3Express Identity Resolver 2024-06-13 14:32:43,352 botocore.utils [DEBUG] Registering S3Express Identity Resolver 2024-06-13 14:32:43,354 botocore.hooks [DEBUG] Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.._handler at 0x000002DB22D50040> 2024-06-13 14:32:43,354 botocore.hooks [DEBUG] Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.._handler at 0x000002DB22D50040> 2024-06-13 14:32:43,356 botocore.hooks [DEBUG] Event creating-resource-class.s3.Object: calling handler <function lazy_call.._handler at 0x000002DB22D500D0> 2024-06-13 14:32:43,356 botocore.hooks [DEBUG] Event creating-resource-class.s3.Object: calling handler <function lazy_call.._handler at 0x000002DB22D500D0> 2024-06-13 14:32:43,358 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x000002DB22C9FE50> 2024-06-13 14:32:43,358 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x000002DB22C9FE50> 2024-06-13 14:32:43,359 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x000002DB22C9FDC0> 2024-06-13 14:32:43,359 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x000002DB22C9FDC0> 2024-06-13 14:32:43,360 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x000002DB22CA6CA0> 2024-06-13 14:32:43,360 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x000002DB22CA6CA0> 2024-06-13 14:32:43,360 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x000002DB23862910>> 2024-06-13 14:32:43,360 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x000002DB23862910>> 2024-06-13 14:32:43,362 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x000002DB23C341C0>> 2024-06-13 14:32:43,362 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x000002DB23C341C0>> 2024-06-13 14:32:43,363 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x000002DB22C9FC10> 2024-06-13 14:32:43,363 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x000002DB22C9FC10> 2024-06-13 14:32:43,363 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x000002DB22CA6E50> 2024-06-13 14:32:43,363 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x000002DB22CA6E50> 2024-06-13 14:32:43,364 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x000002DB23862910>> 2024-06-13 14:32:43,364 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x000002DB23862910>> 2024-06-13 14:32:43,365 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': 'sentinel-s2-l1c', 'Region': 'eu-central-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Key': 'tiles/49/T/EE/2023/7/4/0/B01.jp2', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} 2024-06-13 14:32:43,365 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': 'sentinel-s2-l1c', 'Region': 'eu-central-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Key': 'tiles/49/T/EE/2023/7/4/0/B01.jp2', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} 2024-06-13 14:32:43,366 botocore.regions [DEBUG] Endpoint provider result: https://sentinel-s2-l1c.s3.eu-central-1.amazonaws.com 2024-06-13 14:32:43,366 botocore.regions [DEBUG] Endpoint provider result: https://sentinel-s2-l1c.s3.eu-central-1.amazonaws.com 2024-06-13 14:32:43,367 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-06-13 14:32:43,367 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-06-13 14:32:43,368 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'eu-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True} 2024-06-13 14:32:43,368 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'eu-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True} 2024-06-13 14:32:43,369 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x000002DB22CA4160> 2024-06-13 14:32:43,369 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x000002DB22CA4160> 2024-06-13 14:32:43,369 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x000002DB234BE1F0>> 2024-06-13 14:32:43,369 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x000002DB234BE1F0>> 2024-06-13 14:32:43,370 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function add_recursion_detection_header at 0x000002DB22C9F8B0> 2024-06-13 14:32:43,370 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function add_recursion_detection_header at 0x000002DB22C9F8B0> 2024-06-13 14:32:43,371 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function inject_api_version_header_if_needed at 0x000002DB22CA64C0> 2024-06-13 14:32:43,371 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function inject_api_version_header_if_needed at 0x000002DB22CA64C0> 2024-06-13 14:32:43,372 botocore.endpoint [DEBUG] Making request for OperationModel(name=GetObject) with params: {'url_path': '/tiles/49/T/EE/2023/7/4/0/B01.jp2', 'query_string': {}, 'method': 'GET', 'headers': {'x-amz-request-payer': 'requester', 'User-Agent': 'Boto3/1.34.113 md/Botocore#1.34.113 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.9.16 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.113 Resource'}, 'body': b'', 'auth_path': '/sentinel-s2-l1c/tiles/49/T/EE/2023/7/4/0/B01.jp2', 'url': 'https://sentinel-s2-l1c.s3.eu-central-1.amazonaws.com/tiles/49/T/EE/2023/7/4/0/B01.jp2', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x000002DB23C30F70>, 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'sentinel-s2-l1c', 'params': {'Bucket': 'sentinel-s2-l1c', 'Key': 'tiles/49/T/EE/2023/7/4/0/B01.jp2', 'RequestPayer': 'requester'}}, 'input_params': {'Bucket': 'sentinel-s2-l1c', 'Key': 'tiles/49/T/EE/2023/7/4/0/B01.jp2'}, 'signing': {'region': 'eu-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'eu-central-1'}]}}} 2024-06-13 14:32:43,372 botocore.endpoint [DEBUG] Making request for OperationModel(name=GetObject) with params: {'url_path': '/tiles/49/T/EE/2023/7/4/0/B01.jp2', 'query_string': {}, 'method': 'GET', 'headers': {'x-amz-request-payer': 'requester', 'User-Agent': 'Boto3/1.34.113 md/Botocore#1.34.113 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.9.16 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.113 Resource'}, 'body': b'', 'auth_path': '/sentinel-s2-l1c/tiles/49/T/EE/2023/7/4/0/B01.jp2', 'url': 'https://sentinel-s2-l1c.s3.eu-central-1.amazonaws.com/tiles/49/T/EE/2023/7/4/0/B01.jp2', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x000002DB23C30F70>, 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'sentinel-s2-l1c', 'params': {'Bucket': 'sentinel-s2-l1c', 'Key': 'tiles/49/T/EE/2023/7/4/0/B01.jp2', 'RequestPayer': 'requester'}}, 'input_params': {'Bucket': 'sentinel-s2-l1c', 'Key': 'tiles/49/T/EE/2023/7/4/0/B01.jp2'}, 'signing': {'region': 'eu-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'eu-central-1'}]}}} 2024-06-13 14:32:43,373 botocore.hooks [DEBUG] Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002DB23C30F40>> 2024-06-13 14:32:43,373 botocore.hooks [DEBUG] Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002DB23C30F40>> 2024-06-13 14:32:43,374 botocore.hooks [DEBUG] Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x000002DB22C9FAF0> 2024-06-13 14:32:43,374 botocore.hooks [DEBUG] Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x000002DB22C9FAF0> 2024-06-13 14:32:43,374 botocore.hooks [DEBUG] Event before-sign.s3.GetObject: calling handler <function remove_arn_from_signing_path at 0x000002DB22CA6DC0> 2024-06-13 14:32:43,374 botocore.hooks [DEBUG] Event before-sign.s3.GetObject: calling handler <function remove_arn_from_signing_path at 0x000002DB22CA6DC0> 2024-06-13 14:32:43,376 botocore.hooks [DEBUG] Event before-sign.s3.GetObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x000002DB234BE1F0>> 2024-06-13 14:32:43,376 botocore.hooks [DEBUG] Event before-sign.s3.GetObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x000002DB234BE1F0>> 2024-06-13 14:32:43,377 botocore.auth [DEBUG] Calculating signature using v4 auth. 2024-06-13 14:32:43,377 botocore.auth [DEBUG] Calculating signature using v4 auth. 2024-06-13 14:32:43,378 botocore.auth [DEBUG] CanonicalRequest: GET /tiles/49/T/EE/2023/7/4/0/B01.jp2

host:sentinel-s2-l1c.s3.eu-central-1.amazonaws.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240613T063243Z x-amz-request-payer:requester

host;x-amz-content-sha256;x-amz-date;x-amz-request-payer e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-06-13 14:32:43,378 botocore.auth [DEBUG] CanonicalRequest: GET /tiles/49/T/EE/2023/7/4/0/B01.jp2

host:sentinel-s2-l1c.s3.eu-central-1.amazonaws.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20240613T063243Z x-amz-request-payer:requester

host;x-amz-content-sha256;x-amz-date;x-amz-request-payer e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-06-13 14:32:43,379 botocore.auth [DEBUG] StringToSign: AWS4-HMAC-SHA256 20240613T063243Z 20240613/eu-central-1/s3/aws4_request ce58c486031b91a9636171b213473dd40565e1f1f5235a0043a179ac300fb093 2024-06-13 14:32:43,379 botocore.auth [DEBUG] StringToSign: AWS4-HMAC-SHA256 20240613T063243Z 20240613/eu-central-1/s3/aws4_request ce58c486031b91a9636171b213473dd40565e1f1f5235a0043a179ac300fb093 2024-06-13 14:32:43,380 botocore.auth [DEBUG] Signature: ca5d9b8d9e58373e646e7b641b075678be0d9b808715f1ed1732d6595fda44ca 2024-06-13 14:32:43,380 botocore.auth [DEBUG] Signature: ca5d9b8d9e58373e646e7b641b075678be0d9b808715f1ed1732d6595fda44ca 2024-06-13 14:32:43,381 botocore.hooks [DEBUG] Event request-created.s3.GetObject: calling handler <function add_retry_headers at 0x000002DB22CA6C10> 2024-06-13 14:32:43,381 botocore.hooks [DEBUG] Event request-created.s3.GetObject: calling handler <function add_retry_headers at 0x000002DB22CA6C10> 2024-06-13 14:32:43,382 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sentinel-s2-l1c.s3.eu-central-1.amazonaws.com/tiles/49/T/EE/2023/7/4/0/B01.jp2, headers={'x-amz-request-payer': b'requester', 'User-Agent': b'Boto3/1.34.113 md/Botocore#1.34.113 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.9.16 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.113 Resource', 'X-Amz-Date': b'20240613T063243Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=USERNAME/20240613/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-request-payer, Signature=ca5d9b8d9e58373e646e7b641b075678be0d9b808715f1ed1732d6595fda44ca', 'amz-sdk-invocation-id': b'95592dc5-4bec-417d-b5ae-d3c726a7ec87', 'amz-sdk-request': b'attempt=1'}> 2024-06-13 14:32:43,382 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://sentinel-s2-l1c.s3.eu-central-1.amazonaws.com/tiles/49/T/EE/2023/7/4/0/B01.jp2, headers={'x-amz-request-payer': b'requester', 'User-Agent': b'Boto3/1.34.113 md/Botocore#1.34.113 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.9.16 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.113 Resource', 'X-Amz-Date': b'20240613T063243Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=USERNAME/20240613/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-request-payer, Signature=ca5d9b8d9e58373e646e7b641b075678be0d9b808715f1ed1732d6595fda44ca', 'amz-sdk-invocation-id': b'95592dc5-4bec-417d-b5ae-d3c726a7ec87', 'amz-sdk-request': b'attempt=1'}> 2024-06-13 14:32:43,383 botocore.httpsession [DEBUG] Certificate path: D:\ProgramData\Anaconda3\envs\webserver\lib\site-packages\certifi\cacert.pem 2024-06-13 14:32:43,383 botocore.httpsession [DEBUG] Certificate path: D:\ProgramData\Anaconda3\envs\webserver\lib\site-packages\certifi\cacert.pem 2024-06-13 14:32:43,385 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): sentinel-s2-l1c.s3.eu-central-1.amazonaws.com:443 2024-06-13 14:32:43,385 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): sentinel-s2-l1c.s3.eu-central-1.amazonaws.com:443 2024-06-13 14:32:43.358342 2024-06-13 14:32:44,489 urllib3.connectionpool [DEBUG] https://sentinel-s2-l1c.s3.eu-central-1.amazonaws.com:443 "GET /tiles/49/T/EE/2023/7/4/0/B01.jp2 HTTP/1.1" 200 3064426 2024-06-13 14:32:44,489 urllib3.connectionpool [DEBUG] https://sentinel-s2-l1c.s3.eu-central-1.amazonaws.com:443 "GET /tiles/49/T/EE/2023/7/4/0/B01.jp2 HTTP/1.1" 200 3064426 2024-06-13 14:32:44,491 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': 'eRfOUxPiFiU7Lt0/OSzrNumMSmiBjA/3yXvEzBif4lM9KALMpQrutOURlKokUYBNn8HOF5rT8pQ=', 'x-amz-request-id': 'Q4WDPDZV5EYK73KJ', 'Date': 'Thu, 13 Jun 2024 06:32:46 GMT', 'x-amz-request-charged': 'requester', 'Last-Modified': 'Tue, 04 Jul 2023 08:52:14 GMT', 'ETag': '"6ded918684fdb9895259d1e7bb565b3e"', 'x-amz-storage-class': 'INTELLIGENT_TIERING', 'x-amz-server-side-encryption': 'AES256', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '3064426'} 2024-06-13 14:32:44,491 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': 'eRfOUxPiFiU7Lt0/OSzrNumMSmiBjA/3yXvEzBif4lM9KALMpQrutOURlKokUYBNn8HOF5rT8pQ=', 'x-amz-request-id': 'Q4WDPDZV5EYK73KJ', 'Date': 'Thu, 13 Jun 2024 06:32:46 GMT', 'x-amz-request-charged': 'requester', 'Last-Modified': 'Tue, 04 Jul 2023 08:52:14 GMT', 'ETag': '"6ded918684fdb9895259d1e7bb565b3e"', 'x-amz-storage-class': 'INTELLIGENT_TIERING', 'x-amz-server-side-encryption': 'AES256', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '3064426'} 2024-06-13 14:32:44,492 botocore.parsers [DEBUG] Response body: <botocore.response.StreamingBody object at 0x000002DB234A3A00> 2024-06-13 14:32:44,492 botocore.parsers [DEBUG] Response body: <botocore.response.StreamingBody object at 0x000002DB234A3A00> 2024-06-13 14:32:44,493 botocore.hooks [DEBUG] Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x000002DB238629D0> 2024-06-13 14:32:44,493 botocore.hooks [DEBUG] Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x000002DB238629D0> 2024-06-13 14:32:44,494 botocore.retryhandler [DEBUG] No retry needed. 2024-06-13 14:32:44,494 botocore.retryhandler [DEBUG] No retry needed. 2024-06-13 14:32:44,494 botocore.hooks [DEBUG] Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x000002DB23862910>> 2024-06-13 14:32:44,494 botocore.hooks [DEBUG] Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x000002DB23862910>>`

aws-cli: aws s3 ls s3://sentinel-s2-l1c/tiles/49/T/EE/2023/7/4/0/ --request-payer image this is a folder,aws-cli down this folder is faster,above boto3 python code down a file in the one of this folder(B01.jpg). C:\Users\admin>aws s3 cp s3://sentinel-s2-l1c/tiles/49/T/EE/2023/7/4/0 F:\gbytemp --recursive --request-payer

tim-finnigan commented 1 week ago

Have you tried using download_file documented here? You could also try using multi-processing as documented here. What kind of time differences are you seeing? If you can provide a sample file we can investigate further and try testing on our end.