Failed to insert item - (14k in last 12 hours)

[Ash-Cat]

time=2024-06-21T03:33:06.537Z level=ERROR msg="failed to insert item" username="&{Value: noSmithyDocumentSerde:{}}" key="&{Value:#attribute###https#15220 noSmithyDocumentSerde:{}}" error="operation error DynamoDB: PutItem, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post \"https://dynamodb..amazonaws.com/\": dial tcp: lookup dynamodb..amazonaws.com: i/o timeout"

[noah-tutt-praetorian]

Digital Ocean or EC2?

[Ash-Cat]

1500+ from DO, and 12k+ EC2 approximately

[noah-tutt-praetorian]

We may be hammering dynamoDB (probably not the service itself, but maybe some network component) too hard with goroutines now that the client-side ratelimiting has been removed - I'll test some possible solutions

[Ash-Cat]

For update, last 12 hours, 5686 count

[Ayushi-12]

@Ash-Cat Are we good to close this now?

[Ash-Cat]

We are good to close! No logs showing failed to insert and timeout in last 3 days