Parse logs to JSON before pushing to CloudWatch Logs

[schlarpc]

Currently, logs are parsed just enough to extract timestamps before pushing them to CWL. JSON parsing would enable more effective queries through CW Insights.

Things to think about:

• Testing: this basically requires a stronger test suite than exists today
• Poison pills: how do we deal with mistakes in the parsing logic? push the raw log entry, or drop it on the floor?
• Extra fields: both log formats may gain extra fields in the future, and these must be handled appropriately

[schlarpc]

S3 access logs are impossible to parse robustly.

For instance, if a request's User-Agent contains a double quote, it is unescaped and indistinguishable from a closing quote:

... "aws-cli/1.20.31 Python/3.9.6 Linux/5.10.60.1-microsoft-standard-WSL2 exec-env/[bracket]"quote"'squote' botocore/1.21.31" ...

This might be workable by looking at more fixed elements of the log line, but the S3 docs advise to write a parser that can handle any number of elements:

Occasionally we might extend the access log record format by adding new fields to the end of each line. Therefore, you should write any code that parses server access logs to handle trailing fields that it might not understand.

Since there is no functional escaping, earlier elements in a line can inject values across other elements, leaving the (original) trailing elements to be interpreted as "extensions" at the end of the line.

CloudTrail data events (or EventBridge, which relies on CloudTrail data events) seem like the only robust method to get well-formed S3 access logs, but I'm hesitant to implement this for a few reasons:

• Per event pricing ($0.10 per 100,000 data events delivered)
  • S3 access logging has a cost too, for the PUT request and 1 day of storage, but CloudTrail writes to S3 as well
• Low, fixed quota on CloudTrail trails (5 per region)
  • Can't generally enable account-wide as other buckets in the account could have high traffic, leading to high cost
  • Can't add all buckets that want to opt-in to data events to a single trail with custom resource logic
• Requires pushing to S3 even if you're receiving through CloudWatch Logs or EventBridge