search

aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.34k stars 4.08k forks source link

aws cli cp returns error (404) - missing Key during HeadObject operation after restore-object from Glacier operation #6660

Closed bughunt8 closed 2 years ago

bughunt8 commented 2 years ago

Confirm by changing [ ] to [x] below to ensure that it's a bug:

Describe the bug According to: https://aws.amazon.com/premiumsupport/knowledge-center/restore-s3-object-glacier-storage-class/, the "s3 cp" should be used to overwrite the existing glacier object.

SDK version number % aws --version

aws-cli/2.4.7 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off

Platform/OS/Hardware/Device macOS 10.15.7

To Reproduce (observed behavior)

Assuming the Bucket is bucket1 and Key is /dir1/file1.mp4, then the commands are: (Bucket is Private only, File/Key is in GLACIER)

% aws s3api restore-object --bucket bucket1 --restore-request Days=7,GlacierJobParameters={Tier=Bulk} --key dir1/file1.mp4 % aws s3api head-object --bucket bucket1 --key dir1/file1.mp4

{ "AcceptRanges": "bytes", "Restore": "ongoing-request=\"false\", expiry-date=\"Mon, 24 Jan 2022 00:00:00 GMT\"", "LastModified": "2021-04-29T17:24:04+00:00", "ContentLength": 5335367126, "ETag": "\"xxxxxxxxxxxxxxxxxxxxx\"", "ContentType": "video/mp4", "Metadata": { "mtime": "1604025950" }, "StorageClass": "GLACIER" }

% aws s3 ls s3://bucket11/dir1/

2021-04-30 01:24:04 9264197964 file1.mp4

% aws s3 cp s3://bucket1/dir1/file1.mp4 s3://bucket1/dir1/file1.mp4 --storage-class ONEZONE_IA --debug

Expected behavior A successful copy operation. The same command used to work back in Dec 2021.

Logs/output % aws s3 cp s3://bucket1/dir1/file1.mp4 s3://bucket1/dir1/file1.mp4 --storage-class ONEZONE_IA --force-glacier-transfer --debug

2022-01-17 11:13:16,978 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.4.11 Python/3.8.8 Darwin/19.6.0 exe/x86_64 2022-01-17 11:13:16,978 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', 's3://bucket1/dir1/file1.mp4', 's3://bucket1/dir1/file1.mp4', '--storage-class', 'ONEZONE_IA', '--force-glacier-transfer', '--debug'] 2022-01-17 11:13:17,028 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7ffea12670d0> 2022-01-17 11:13:17,028 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7ffea1140700> 2022-01-17 11:13:17,028 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>> 2022-01-17 11:13:17,028 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7ffea10d5e50> 2022-01-17 11:13:17,028 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7ffea10e4e50> 2022-01-17 11:13:17,028 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7ffe880f5af0> 2022-01-17 11:13:17,028 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7ffe901aa4c0> 2022-01-17 11:13:17,029 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>> 2022-01-17 11:13:17,029 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7ffea126cd30> 2022-01-17 11:13:17,030 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/data/cli.json 2022-01-17 11:13:17,033 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7ffea11f3f70> 2022-01-17 11:13:17,033 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7ffea11fbaf0> 2022-01-17 11:13:17,033 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7ffea11fba60> 2022-01-17 11:13:17,033 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7ffea11fbc10> 2022-01-17 11:13:17,033 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7ffea11fbb80> 2022-01-17 11:13:17,033 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7ffe804d38c0> 2022-01-17 11:13:17,034 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.4.11 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off 2022-01-17 11:13:17,034 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', 's3://bucket1/dir1/file1.mp4', 's3://bucket1/dir1/file1.mp4', '--storage-class', 'ONEZONE_IA', '--force-glacier-transfer', '--debug'] 2022-01-17 11:13:17,034 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7ffea1267700> 2022-01-17 11:13:17,034 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7ffea0ecc5e0> 2022-01-17 11:13:17,034 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7ffea1299c10> 2022-01-17 11:13:17,035 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7ffea0ec6a60> 2022-01-17 11:13:17,035 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7ffea0f3df70> 2022-01-17 11:13:17,038 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/ 2022-01-17 11:13:17,044 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7ffe901aa3a0> 2022-01-17 11:13:17,044 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7ffea113a5e0> 2022-01-17 11:13:17,044 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7ffea126cd30> 2022-01-17 11:13:17,044 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,045 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_cp: calling handler <function add_waiters at 0x7ffea126cd30> 2022-01-17 11:13:17,047 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,049 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,049 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ffea0f0aca0> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ffea0f0aca0> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ffea0f0aca0> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,050 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ffea0f0aca0> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,051 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ffea0f0aca0> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,053 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ffea0f0aca0> 2022-01-17 11:13:17,053 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.force-glacier-transfer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,053 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ffea0f0aca0> 2022-01-17 11:13:17,053 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,053 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,053 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.copy-props: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,053 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ffea0f0aca0> 2022-01-17 11:13:17,053 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,054 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,054 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ffea12dbf70> 2022-01-17 11:13:17,054 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env 2022-01-17 11:13:17,054 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role 2022-01-17 11:13:17,054 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity 2022-01-17 11:13:17,054 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso 2022-01-17 11:13:17,054 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file 2022-01-17 11:13:17,055 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials 2022-01-17 11:13:17,057 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/endpoints.json 2022-01-17 11:13:17,072 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ffea09fbb80> 2022-01-17 11:13:17,107 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/service-2.json 2022-01-17 11:13:17,118 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ffea0964310> 2022-01-17 11:13:17,118 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ffea09640d0> 2022-01-17 11:13:17,202 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60) 2022-01-17 11:13:17,206 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ffea09fbb80> 2022-01-17 11:13:17,207 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ffea0964310> 2022-01-17 11:13:17,207 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ffea09640d0> 2022-01-17 11:13:17,211 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60) 2022-01-17 11:13:17,212 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ffea09fbb80> 2022-01-17 11:13:17,213 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7ffea0964310> 2022-01-17 11:13:17,213 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7ffea09640d0> 2022-01-17 11:13:17,216 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60) 2022-01-17 11:13:17,217 - MainThread - awscli.customizations.s3.factory - DEBUG - Using a multipart threshold of 5368709120 and a part size of 8388608 2022-01-17 11:13:17,218 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.HeadObject: calling handler <function base64_decode_input_blobs at 0x7ffea129c3a0> 2022-01-17 11:13:17,219 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7ffea0a1cdc0> 2022-01-17 11:13:17,219 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7ffea0a1cd30> 2022-01-17 11:13:17,219 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7ffea13344c0>> 2022-01-17 11:13:17,219 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7ffea1334580>> 2022-01-17 11:13:17,219 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7ffea0a1cb80> 2022-01-17 11:13:17,219 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7ffea0a220d0> 2022-01-17 11:13:17,220 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7ffea13344c0>> 2022-01-17 11:13:17,220 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function inject_api_version_header_if_needed at 0x7ffea0a23430> 2022-01-17 11:13:17,220 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=HeadObject) with params: {'url_path': '/bucket1/dir1/file1.mp4', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'aws-cli/2.4.11 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off command/s3.cp'}, 'body': b'', 'url': 'https://s3.ap-east-1.amazonaws.com/bucket1/dir1/file1.mp4', 'context': {'client_region': 'ap-east-1', 'client_config': <botocore.config.Config object at 0x7ffea12f5970>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'bucket1'}}} 2022-01-17 11:13:17,223 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ffea12f5910>> 2022-01-17 11:13:17,223 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7ffea1334610>> 2022-01-17 11:13:17,223 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7ffea0a1ca60> 2022-01-17 11:13:17,223 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7ffea1334610>> 2022-01-17 11:13:17,224 - MainThread - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback. 2022-01-17 11:13:17,224 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.ap-east-1.amazonaws.com/bucket1/dir1/file1.mp4 2022-01-17 11:13:17,224 - MainThread - botocore.utils - DEBUG - URI updated to: https://bucket1.s3.ap-east-1.amazonaws.com/dir1/file1.mp4 2022-01-17 11:13:17,224 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth. 2022-01-17 11:13:17,224 - MainThread - botocore.auth - DEBUG - CanonicalRequest: HEAD /dir1/file1.mp4 host:bucket1.s3.ap-east-1.amazonaws.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20220117T031317Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2022-01-17 11:13:17,224 - MainThread - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20220117T031317Z 20220117/ap-east-1/s3/aws4_request 3976e8f601a16346c7685fe372964f0fd35ce0b5208608bd18486df417e76226 2022-01-17 11:13:17,225 - MainThread - botocore.auth - DEBUG - Signature: db3126e3ecbae87fb09868fc752c9855daec46f859c83c775180a1e621c78dc4 2022-01-17 11:13:17,225 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://bucket1.s3.ap-east-1.amazonaws.com/dir1/file1.mp4, headers={'User-Agent': b'aws-cli/2.4.11 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off command/s3.cp', 'X-Amz-Date': b'20220117T031317Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAQZQ24UV4VJCAKMVI/20220117/ap-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=db3126e3ecbae87fb09868fc752c9855daec46f859c83c775180a1e621c78dc4'}> 2022-01-17 11:13:17,226 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem 2022-01-17 11:13:17,226 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ncp1-rclone1.s3.ap-east-1.amazonaws.com:443 2022-01-17 11:13:17,353 - MainThread - urllib3.connectionpool - DEBUG - https://bucket1.s3.ap-east-1.amazonaws.com:443 "HEAD /dir1/file1.mp4 HTTP/1.1" 404 0 2022-01-17 11:13:17,353 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': 'GMJVVVMDVN6RE2AZ', 'x-amz-id-2': 'aS1se6X/p/v0suwkmaBNi1SgsKbjB3ln3yU4GfO2oVBXkrDVPdKa2Ta0HpP81BBe0B9PSW0zK4k=', 'Content-Type': 'application/xml', 'Date': 'Mon, 17 Jan 2022 03:13:16 GMT', 'Server': 'AmazonS3'} 2022-01-17 11:13:17,354 - MainThread - botocore.parsers - DEBUG - Response body: b'' 2022-01-17 11:13:17,358 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7ffea1334460>> 2022-01-17 11:13:17,358 - MainThread - botocore.retries.standard - DEBUG - Not retrying request. 2022-01-17 11:13:17,358 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7ffea13344c0>> 2022-01-17 11:13:17,358 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x7ffea1267310> 2022-01-17 11:13:17,358 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7ffea12f5f70>> 2022-01-17 11:13:17,359 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: An error occurred (404) when calling the HeadObject operation: Key "dir1/file1.mp4" does not exist Traceback (most recent call last): File "awscli/customizations/s3/filegenerator.py", line 355, in _list_single_object File "awscli/botocore/client.py", line 281, in _api_call File "awscli/botocore/client.py", line 609, in _make_api_call botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/customizations/s3/s3handler.py", line 149, in call File "awscli/customizations/s3/fileinfobuilder.py", line 31, in call File "awscli/customizations/s3/filegenerator.py", line 142, in call File "awscli/customizations/s3/filegenerator.py", line 318, in list_objects File "awscli/customizations/s3/filegenerator.py", line 366, in _list_single_object botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Key "dir1/file1.mp4" does not exist fatal error: An error occurred (404) when calling the HeadObject operation: Key "dir1/file1.mp4" does not exist 2022-01-17 11:13:17,359 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

bughunt8 commented 2 years ago

After retrying this - this seemed to have resolved. Closing this for now.

github-actions[bot] commented 2 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.