hwine
commented
5 years ago Another example, which definitely needs a retry (connection lost):
2019-09-09T13:50:06.861Z START RequestId: cc48bd72-842e-45a2-8053-d9edeaa3a5a7 Version: $LATEST
2019-09-09T13:50:07.598Z {"input_event": {"Records": [{"EventVersion": "1.0", "EventSubscriptionArn": "arn:aws:sns:us-east-1:361527076523:productdelivery-prod-s3events:22bd6044-7fc0-4695-9471-02f9c2e70896", "EventSource": "aws:sns", "Sns": {"SignatureVersion": "1", "Timestamp": "2019-09-09T13:50:06.801Z", "Signature": "P/lJA7JTZys3izbWVU9o+y66t7R0cYuSN0bBFxe+YWu4vxErMIQ90xL+BkQnKjX9QK8zGQH2dJH9JlZx4F99y57V+8Du8kYt5wk48EWBhtusv26zUNGyCCOPB2bqLLhrAoevRzqR4hGG19vFpj7RrHU4UFDS7U+GdemGskJMq8M1abQfSfY9QJevvzcSVdxZ6PTCko5b/8kYOCwNCreiCagFkkO9qM5QokZBw6h6gbiKfDDXacimKvU7Z5jpXstt7BeEkg2zY2YHMgidDsE7yx4IuVPmTzLcm8W4UrtzygERWSRAvxNMrt67qc/fHtDAmHF9gkDWNNbT/Av4RDDHlg==", "SigningCertUrl": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-6aad65c2f9911b05cd53efda11f913f9.pem", "MessageId": "46a6151c-144e-5deb-904d-40ec4e80fb79", "Message": "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2019-09-09T13:50:06.237Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AWS:AIDAIZ6TDWSYXJXAGPXTY\"},\"requestParameters\":{\"sourceIPAddress\":\"34.83.21.167\"},\"responseElements\":{\"x-amz-request-id\":\"27B6143F8D9C7098\",\"x-amz-id-2\":\"dMg/Pag0EpMYCR0eewk0OJBQ1hKfgeUA40rtqkCZc3bqFfSUK04dBhpbtg2Q8ejtPP6aQKf2C2c=\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"CreationEvents\",\"bucket\":{\"name\":\"net-mozaws-prod-delivery-firefox\",\"ownerIdentity\":{\"principalId\":\"A85MX00GSLCB4\"},\"arn\":\"arn:aws:s3:::net-mozaws-prod-delivery-firefox\"},\"object\":{\"key\":\"pub/firefox/nightly/latest-mozilla-central-l10n/firefox-71.0a1.son.win32.installer.exe\",\"size\":51048528,\"eTag\":\"c0f606feb493ae61b4cf640bcfdeb8f2\",\"versionId\":\"PLKn2.7htUNqNaZSreNgNm83oPUqWxYz\",\"sequencer\":\"005D76588A0C48D376\"}}}]}", "MessageAttributes": {}, "Type": "Notification", "UnsubscribeUrl": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:361527076523:productdelivery-prod-s3events:22bd6044-7fc0-4695-9471-02f9c2e70896", "TopicArn": "arn:aws:sns:us-east-1:361527076523:productdelivery-prod-s3events", "Subject": "Amazon S3 Notification"}}]}, "version": "0.4.9", "unpacker": "SNS message unpacked", "results": [{"status": "fail", "bucket": "net-mozaws-prod-delivery-firefox", "results": ["failed to process s3 object net-mozaws-prod-delivery-firefox/pub/firefox/nightly/latest-mozilla-central-l10n/firefox-71.0a1.son.win32.installer.exe 'ProtocolError(\"Connection broken: error(104, 'Connection reset by peer')\", error(104, 'Connection reset by peer'))'"], "key": "pub/firefox/nightly/latest-mozilla-central-l10n/firefox-71.0a1.son.win32.installer.exe", "s3wait": 0.1927340030670166}], "request_id": "cc48bd72-842e-45a2-8053-d9edeaa3a5a7"}
2019-09-09T13:50:07.598Z END RequestId: cc48bd72-842e-45a2-8053-d9edeaa3a5a7
2019-09-09T13:50:07.598Z REPORT RequestId: cc48bd72-842e-45a2-8053-d9edeaa3a5a7 Duration: 734.96 ms Billed Duration: 800 ms Memory Size: 1024 MB Max Memory Used: 204 MB
GH-60 shows that not all S3 failures are retried. When this happens, an artifact does not undergo verification.
There are two likely candidates for this:
So far, alert volume has been very low - just the one instance in #60. Thus giving only medium priority. If frequency increases, this should be high priority.