Open dsotirho-ucsc opened 1 year ago
Spike to investigate and diagnose.
It was very difficult to correlate the failures to specific log files. Perhaps we should consider finding a way to streamline that step.
The 7 log files that failed to forward are
2023-09-30-06-27-54-649044166FC4297A
2023-09-30-06-40-04-84E35196EF38F3BE
2023-09-30-06-28-15-19BBF71A613294D1
2023-09-30-06-20-12-DCA8C497A9E01293
2023-09-30-06-14-53-ADDCA5BBA34C010A
2023-09-30-06-13-59-A61E34742158854B
2023-09-30-06-26-12-8D2FC67E2A0AD99D
in folder s3://edu-ucsc-gi-platform-hca-prod-logs.us-east-1/s3/access/.data-browser/org-humancellatlas-data-portal-dcp2-prod/
Here is how the contents of the 1st such file would/should have been forwarded if the decoding error had not occurred:
{
"bucket_owner": "37a70605c2e04b2d81c08054e4646a531acd4bff3e16d9f65b3cff2090399cdd",
"bucket": "org-humancellatlas-data-portal-dcp2-prod",
"remote_ip": "130.176.93.86",
"requester": "-",
"request_id": "21BDJ4WAGRPA7XN4",
"operation": "REST.HEAD.OBJECT",
"key": "%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F%253F.tar.gz",
"request_uri": "HEAD /����.tar.gz HTTP/1.1",
"http_status": "403",
"error_code": "SignatureDoesNotMatch",
"bytes_sent": "7350",
"object_size": "-",
"total_time": "7",
"turn_around_time": "-",
"referer": "-",
"user_agent": "Amazon CloudFront",
"version_id": "-",
"host_id": "zHQ0T9q8qgZStZrzQQtCGKTKlHfjmHe0S3/ZLdjvovIQmMX3m9MXKC/F+G7F5mglKhNVK9efV2Qew+EVcZ2YJw==",
"signature_version": "SigV4",
"cipher_suite": "ECDHE-RSA-AES128-GCM-SHA256",
"authentication_type": "AuthHeader",
"host_header": "org-humancellatlas-data-portal-dcp2-prod.s3.us-east-1.amazonaws.com",
"tls_version": "TLSv1.2",
"access_point_arn": "-",
"acl_required": "-",
"time": "30/Sep/2023:05:54:35 +0000"
}
So, it appears that the S3 logs apply URL-encoding to the object key in the key
field, but not in the request_uri
field, resulting in the presence of non-ASCII characters.
Switch to latin1
codec for both S3 access and ALB log forwarders.
@hannes-ucsc: "De-prioritizing due to reduced prevalence. Probably because WAF is catching the requests that trigger this."
Occurred 7 times within 25 minutes on 2023/09/29 Link to CloudWatch logs)