tim-finnigan
commented
9 months ago Hi @sdarwin thanks for reaching out. Here is the delete-objects documentation for reference: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-objects.html.
The 1000 limit you referenced is mentioned there:
The request contains a list of up to 1000 keys that you want to delete
I could reproduce the MalformedXML error using a file containing 1001 keys generated by this script:
import json
data = {
"Objects": [
{
"Key":f"Key_{i}",
} for i in range(1, 1002)
],
"Quiet": True
}
with open('data.json', 'w') as outfile:
json.dump(data, outfile, indent=4)aws s3api delete-objects --bucket $BUCKET --delete file://data.json resulted in:
An error occurred (MalformedXML) when calling the DeleteObjects operation: The XML you provided was not well-formed or did not validate against our published schema(The SSL validation failed... error could have occurred for various reasons. Those errors are often caused by certificate issues as noted in the troubleshooting guide. But if it was something that you just encountered once then it may have been some kind of transient network/proxy issue.)
I also think that the error message could be improved. That error message is returned by the S3 DeleteObjects API, so we'll need to reach out to the S3 team to request a better error message. I'll go ahead and transfer this to our cross-SDK repository for further tracking as service APIs like this are used across SDKs. I'll also note a few related issues I found:
sdarwin
Describe the feature
Improve the accuracy of the error messages during certain failures.
Use Case
Clear error messages assist in debugging.
Here is the problem I encountered. When running this command to delete objects in S3:
It supposedly has a limit of 1000 items. If you provide a million items to delete, the error messages are mysterious:
or
As evidence for MalformedXML error, consider this stackexchange article
"+1 for number of keys. I did not check the go code but for us in python boto3 over 1000 keys in the request caused the error. You would think they would document this is the documentation for MalformedXML on the sdk client, but you did find api documentation. We chunked the keys and it works perfect – vfrank66"
Proposed Solution
The problem is that neither message explains the actual cause of the problem, which is the input is over 1000 items in length. The errors should be more specific about the real reason.
Other Information
No response
Acknowledgements
CLI version used
aws-cli/2.13.17 Python/3.11.5 Linux/6.2.0-1013-gcp exe/x86_64.ubuntu.22 prompt/off
Environment details (OS name and version, etc.)
Ubuntu 22.04