The S3 API may reject requests with a custom error (503 Slow Down) when a rate
limit has been reached. Boto automatically retries these requests, this does
not always work. Example traceback:
Traceback (most recent call last):
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/boto3/s3/transfer.py", line 279, in upload_file
future.result()
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/futures.py", line 233, in result
raise self._exception
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 6, in <module>
cli.main()
File "/home/username/pxl/pxl/cli.py", line 224, in main
cli()
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/username/pxl/pxl/cli.py", line 127, in upload_cmd
image = upload.public_image_with_size(client, entry)
File "/home/username/pxl/pxl/upload.py", line 102, in public_image_with_size
public_image(client, local_filename, object_name)
File "/home/username/pxl/pxl/upload.py", line 124, in public_image
Key=object_name,
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/boto3/s3/inject.py", line 131, in upload_file
extra_args=ExtraArgs, callback=Callback)
File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/boto3/s3/transfer.py", line 287, in upload_file
filename, '/'.join([bucket, key]), e))
boto3.exceptions.S3UploadFailedError: Failed to upload /tmp/photofile.jpeg to the-bucket-name/abcdef12-3456-7890-abcd-ef1234567890_w_1600.jpg: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.
The S3 API may reject requests with a custom error (503 Slow Down) when a rate limit has been reached. Boto automatically retries these requests, this does not always work. Example traceback:
Some sort of backoff handling would probably be useful. I found https://pypi.org/project/backoff/, which would probably work.