tim-finnigan
commented
1 week ago Thanks for reaching out. We do have a troubleshooting section on SSL Validation errors, which mentions certificate issues as being a common cause, but in this case the error does seem related to the huge file size.
Can you provide your full debug logs (with sensitive info redacted) by adding --debug to the command? That could help give more insight into what's going on here.
A few other suggestions:
- Migrate to use v2: https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html
- Enable S3 Transfer Acceleration: https://aws.amazon.com/s3/transfer-acceleration/
- Use the CRT: https://aws.amazon.com/blogs/storage/improving-amazon-s3-throughput-for-the-aws-cli-and-boto3-with-the-aws-common-runtime/
- Increase the
--cli-connect-timeout, documented here: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-options.html - Try different S3 configurations, documented here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html
- Try using the low-level
s3apiclient to do a multipart upload: https://repost.aws/knowledge-center/s3-multipart-upload-cli
Please let us know what you have tried and the results, and we can review your logs as well once you provide those.
github-actions[bot]
Describe the bug
I am currently trying to use the AWS CLI to upload a large 500GB zip file stored on my external hard drive as a single object to S3. I leave the process running for a few hours, where it continues just fine, but before the upload can complete, the console throws an error.
This error only gets thrown when I'm uploading a large file. When I upload a small 1GB file, the upload completes successfully with no issues or errors
Expected Behavior
I expect the console to return no errors and the zip file to show up in my bucket as a successfully uploaded object.
Current Behavior
What currently happens is after about 8 hours of going through the upload process, the console throws the following error:
upload failed: 'localPathToZipFile' to s3://bucket-name/zipFileName.zip SSL validation failed for https://bucket-name.s3.us-east-1.amazonaws.com/bucket-name/zipFileName.zip?uploadId=aegU_bhqF9utYLQZDEuwfrgtdgVgf.ccRFn03uicuS7r5qzolOi9cbeqm5TrvPgjz4sCOfdAevC.Sboj3UQH577T9D4be8pg1BxDC.ori15PXNIXptkk9tZDVuVyhCPd&partNumber=1373 EOF occurred in violation of protocol (_ssl.c:2406)Reproduction Steps
I run the following command:
aws s3 cp localPathToZipFile s3://bucket-name --endpoint-url https://bucket-name.s3.us-east-1.amazonaws.comPossible Solution
I read online that the reason I'm getting this error is because my SSL certificate isn't configured correctly. As a test, I ran the same command as above, but with --no-verify-ssl added. (I'm aware that I should never actually do this. It's very unsecure. I'm just testing to see what's wrong.)
I run the following command:
aws s3 cp localPathToZipFile s3://bucket-name --endpoint-url https://bucket-name.s3.us-east-1.amazonaws.com --no-verify-sslBut I STILL get the same error thrown as before. I'm not sure what could be causing this error. Could it be some kind of proxy or network issue? I checked to make sure my VPN was disabled.
Additional Information/Context
As mentioned above, this issue only occurs when I upload a large 500GB zip file on my external hard drive. Any small files that are a few GBs get uploaded with no issues or errors.
The process runs fine for a few hours, but throws the error in the middle of the upload.
CLI version used
aws-cli/1.33.8 Python/3.12.0 Darwin/21.6.0 botocore/1.34.126
Environment details (OS name and version, etc.)
macOS Monterey 12.6.5