cfurst
commented
4 years ago This also happens when copying object from bucket to bucket which according to the docs is the way to move files out of the glacier tier.
Closed cfurst closed 3 years ago
cfurst
commented
4 years ago This also happens when copying object from bucket to bucket which according to the docs is the way to move files out of the glacier tier.
cfurst
commented
4 years ago However when I copy individual objects down from s3 using the cli, they copy fine, whether they are Glacier objects or not.
cfurst
commented
4 years ago for example here is a head-object call from the cli on VAAS/pom.xml
{
"AcceptRanges": "bytes",
"Restore": "ongoing-request=\"false\", expiry-date=\"Thu, 18 Jun 2020 00:00:00 GMT\"",
"LastModified": "2018-09-13T05:41:54+00:00",
"ContentLength": 1068,
"ETag": "\"61c8ace1151b74f8bce7b77474617dc0\"",
"ContentType": "application/xml",
"Metadata": {},
"StorageClass": "GLACIER"
}Notice the restore field says that it has been restored and should be accessable until the 18th of June GMT (June 17th 20:00:00 local time). These requests happened on the 4th.
KaibaLopez
commented
4 years ago Hi @cfurst ,
So, because glacier's restore nature this will take me a while to replicate, in the mean time though, can you try adding these options to your cli command calls?
--ignore-glacier-warnings
Or
--force-glacier-transfer
Let me know if those worked for you, it does seem weird to me that the single ones are working, but I won't know more until I test it.
Also, can you show me the command you're using on CLI? I'm assuming you're using aws s3 cp, but I don't think I saw it on your previous comments and it'd be better if I can know exactly what you are trying to do (no private info tho).
cfurst
commented
4 years ago Hey @KaibaLopez
Thanks for the tips, I'll give those a try! In the meantime here are some of the aws cli calls I made regarding the above:
Copy restored folder to local (this is in PowerShell):
aws --debug s3 cp --recursive s3://cfurst-mybooklive-bkup/VAAS .
Sync:
aws --debug s3 sync s3://cfurst-mybooklive-bkup/VAAS .
Bucket to bucket:
aws s3 cp --recursive s3://cfurst-mybooklive-bkup/VASS s3://cfurst-mybooklive-restore
This is a file I copied locally that was never restored:
aws s3 cp "s3://cfurst-mybooklive-bkup/2013 gift cards - Thanksgiving/Amazon-Gift-Card-anne.pdf" .
Head object on the above:
aws s3api head-object --bucket cfurst-mybooklive-bkup --key "2013 gift cards - Thanksgiving/Amazon-Gift-Card-anne.pdf"
{
"AcceptRanges": "bytes",
"Restore": "ongoing-request=\"false\", expiry-date=\"Thu, 18 Jun 2020 00:00:00 GMT\"",
"LastModified": "2018-09-13T03:59:35+00:00",
"ContentLength": 216536,
"ETag": "\"c8fe8d7897448d1dbeba8884c9f1e6cf\"",
"ContentType": "application/pdf",
"Metadata": {},
"StorageClass": "GLACIER"
}and here is the debug version of the copy without the auth stuff and other HEAD calls etc..
PS C:\Users\Carl Yamamoto-Furst> aws --debug s3 cp "s3://cfurst-mybooklive-bkup/2013 gift cards - Thanksgiving/Amazon-Gift-Card-anne.pdf" .
...
2020-06-07 20:59:25,681 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest:
GET
/2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf
host:cfurst-mybooklive-bkup.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200608T005925Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-06-07 20:59:25,687 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20200608T005925Z
20200608/us-east-1/s3/aws4_request
302add85aeb6c3121f12e387a8099c18fda1956804be3467435399fbc1165d3f
2020-06-07 20:59:25,689 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Signature:
c8f763e11b098197356d025fd08da07c80ada03539daa64af7ae5d4a1fdb1a84
2020-06-07 20:59:25,690 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x0000015BF36E49D8>
2020-06-07 20:59:25,691 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://cfurst-mybooklive-bkup.s3.amazonaws.com/2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf, headers={'User-Agent': b'aws-cli/2.0.17 Python/3.7.7 Windows/10 botocore/2.0.0dev21', 'X-Amz-Date': b'20200608T005925Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256...'}>
2020-06-07 20:59:25,693 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): cfurst-mybooklive-bkup.s3.amazonaws.com:443
2020-06-07 20:59:26,015 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - https://cfurst-mybooklive-bkup.s3.amazonaws.com:443 "GET /2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf HTTP/1.1" 400 None
2020-06-07 20:59:26,017 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': '2439F8140BDCE931', 'x-amz-id-2': '2SJdkmz0bGXLXRsInpPOJJ83LCzPnZP+MRF+DeTkNNE2/wdtq6CzCV8PGSWp8VqI0pwWVYxqxV0=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Mon, 08 Jun 2020 00:59:25 GMT', 'Connection': 'close', 'Server': 'AmazonS3'}
2020-06-07 20:59:26,020 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \'us-east-1\' is wrong; expecting \'us-west-2\'</Message><Region>us-west-2</Region><RequestId>2439F8140BDCE931</RequestId><HostId>2SJdkmz0bGXLXRsInpPOJJ83LCzPnZP+MRF+DeTkNNE2/wdtq6CzCV8PGSWp8VqI0pwWVYxqxV0=</HostId></Error>'
2020-06-07 20:59:26,028 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x0000015BF4867948>>
2020-06-07 20:59:26,029 - ThreadPoolExecutor-0_0 - botocore.retries.standard - DEBUG - Not retrying request.
2020-06-07 20:59:26,033 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x0000015BF4867508>>
2020-06-07 20:59:26,035 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - S3 client configured for region aws-global but the bucket cfurst-mybooklive-bkup is in region us-west-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
2020-06-07 20:59:26,036 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - Updating URI from https://s3.amazonaws.com/cfurst-mybooklive-bkup/2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf to https://s3.us-west-2.amazonaws.com/cfurst-mybooklive-bkup/2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf
2020-06-07 20:59:26,037 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0 seconds
2020-06-07 20:59:26,039 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x0000015BF36E4948>
2020-06-07 20:59:26,040 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000015BF4815748>>
2020-06-07 20:59:26,041 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x0000015BF30EEAF8>
2020-06-07 20:59:26,046 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x0000015BF4867C48>>
2020-06-07 20:59:26,048 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.us-west-2.amazonaws.com/cfurst-mybooklive-bkup/2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf
2020-06-07 20:59:26,049 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - URI updated to: https://cfurst-mybooklive-bkup.s3.us-west-2.amazonaws.com/2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf
2020-06-07 20:59:26,051 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2020-06-07 20:59:26,052 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest:
GET
/2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf
host:cfurst-mybooklive-bkup.s3.us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200608T005926Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-06-07 20:59:26,053 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20200608T005926Z
20200608/us-west-2/s3/aws4_request
bc7cdc9ae5d176afc3fa8ffca7aebcd9b927f622128249e3481ae0a5db8beabb
2020-06-07 20:59:26,059 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Signature:
bbb3578b1035e914b2f2ffe6568dcdf4a3d514812e1b58ff418ec31546e3fe64
2020-06-07 20:59:26,060 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x0000015BF36E49D8>
2020-06-07 20:59:26,062 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://cfurst-mybooklive-bkup.s3.us-west-2.amazonaws.com/2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf, headers={'User-Agent': b'aws-cli/2.0.17 Python/3.7.7 Windows/10 botocore/2.0.0dev21', 'X-Amz-Date': b'20200608T005926Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 ...'}>
2020-06-07 20:59:26,064 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): cfurst-mybooklive-bkup.s3.us-west-2.amazonaws.com:443
2020-06-07 20:59:26,496 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - https://cfurst-mybooklive-bkup.s3.us-west-2.amazonaws.com:443 "GET /2013%20gift%20cards%20-%20Thanksgiving/Amazon-Gift-Card-anne.pdf HTTP/1.1" 200 216536
2020-06-07 20:59:26,497 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '/UEe3yUXXjJps1aVxfTEtkFLpf6dJ2E+6s9Coufq1WWqSlOX+5UDld8Q20/MDDVOpisfe1HNf9g=', 'x-amz-request-id': 'FFF7EFF0DC213E4D', 'Date': 'Mon, 08 Jun 2020 00:59:26 GMT', 'Last-Modified': 'Thu, 13 Sep 2018 03:59:35 GMT', 'x-amz-restore': 'ongoing-request="false", expiry-date="Thu, 18 Jun 2020 00:00:00 GMT"', 'ETag': '"c8fe8d7897448d1dbeba8884c9f1e6cf"', 'x-amz-storage-class': 'GLACIER', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/pdf', 'Content-Length': '216536', 'Server': 'AmazonS3'}
2020-06-07 20:59:26,498 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response body:
<botocore.response.StreamingBody object at 0x0000015BF4B47308>
2020-06-07 20:59:26,504 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x0000015BF4867948>>
2020-06-07 20:59:26,505 - ThreadPoolExecutor-0_0 - botocore.retries.standard - DEBUG - Not retrying request.
2020-06-07 20:59:26,506 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x0000015BF4867508>>
2020-06-07 20:59:26,507 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - S3 request was previously redirected, not redirecting.
2020-06-07 20:59:26,508 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <function enhance_error_msg at 0x0000015BF44BB288>
2020-06-07 20:59:26,510 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x0000015BF4867348>>
2020-06-07 20:59:26,761 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
2020-06-07 20:59:26,762 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
2020-06-07 20:59:26,765 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x0000015BF4AC8808>, 'offset': 0}
2020-06-07 20:59:26,768 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': 'C:\\Users\\Carl Yamamoto-Furst\\Amazon-Gift-Card-anne.pdf'}) about to wait for the following futures []
2020-06-07 20:59:26,769 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': 'C:\\Users\\Carl Yamamoto-Furst\\Amazon-Gift-Card-anne.pdf'}) done waiting for dependent futures
2020-06-07 20:59:26,771 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IORenameFileTask(transfer_id=0, {'final_filename': 'C:\\Users\\Carl Yamamoto-Furst\\Amazon-Gift-Card-anne.pdf'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x0000015BF4AC8808>, 'final_filename': 'C:\\Users\\Carl Yamamoto-Furst\\Amazon-Gift-Card-anne.pdf', 'osutil': <s3transfer.utils.OSUtils object at 0x0000015BF4AC2488>}
2020-06-07 20:59:26,863 - ThreadPoolExecutor-0_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download: s3://cfurst-mybooklive-bkup/2013 gift cards - Thanksgiving/Amazon-Gift-Card-anne.pdf to .\Amazon-Gift-Card-anne.pdf
2020-06-07 20:59:26,868 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.
PS C:\Users\Carl Yamamoto-Furst>Upon further examination it looks like the Thanksgiving file I downloaded seems to have been restored. I don't recall restoring it, but that makes the case where downloading a single file can be done even though it's in Glacier false. Let me see if I can reproduce with something the console says is still in Glacier and not restored.
Sorry for the confusion.
cfurst
commented
4 years ago OK confimed that the single file case was false:
PS C:\Users\Carl Yamamoto-Furst> aws s3 cp "s3://cfurst-mybooklive-bkup/Downloads/PallasadesStatePark.pdf.pdf" .
warning: Skipping file s3://cfurst-mybooklive-bkup/Downloads/PallasadesStatePark.pdf.pdf. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 download help for additional parameter options to ignore or force these transfers.This file was reported in the console as not being restored and yep, it can't be downloaded. So we can ignore that case for now and focus on recursive copying of restored files.
Again, apologies for the confusion.
KaibaLopez
commented
4 years ago Hi @cfurst , I've dug around a bit, and I've got some bad news. This seems to be a service limitation, which means that it is not something we can fix directly from the CLI... I'll keep it as a Feature request and escalate this, but yea, keep in mind this will probably take a while. And I guess the best workaround I can think of for now is trying single downloads. Sorry I couldn't be more helpful and thanks for bringing this up to us.
cfurst
commented
4 years ago Hi @KaibaLopez ! I apologize for the late replay I tried to use --force-glacier-transfer and it worked! Thanks for the tip, but I'm curious, can you tell me what that does? Does that force the transfer of a Glacier object whether or not the restore is in place? Do you pay more for that kind of transfer?
kdaily
commented
3 years ago Hi @cfurst, this closed issue should give you the context regarding what the parameter does, as it looks like the same thing prompted the addition of the --force-glacier-transfer parameter:
https://github.com/aws/aws-cli/issues/1699
I do not believe that there is any difference in cost once an object is restored from Glacier.
Confirm by changing [ ] to [x] below to ensure that it's a bug:
Describe the bug After requesting a file restore via the python sdk, confirming in the console that objects are temporarily restored, the aws cli throws errors saying the objects are being skipped because they are in Glacier when doing a sync or a recursive copy down onto local machine
SDK version number aws-cli/2.0.17 Python/3.7.7 Windows/10 botocore/2.0.0dev21
Platform/OS/Hardware/Device Windows 10 on a dell Lattitude
To Reproduce (observed behavior) place a lifecycle on a bucket that put objects in Glacier storage tier after a specific number of days. Use the boto3 python sdk to make a bulk file restore from glacier. This is assuming you have a number of files in a folder in s3. After the restore is complete, try to recursively copy the folder down, or try to do a sync of the folder to the local machine.
Expected behavior The restored files should be downloaded and not skipped
Logs/output
Additional context All the files above have a temporary restore in place.. that expires June 17th.