Another AWS log format that would be very useful to process with a Glue Job and query with Athena is AWS WAF logs. Today we can save it to an ElasticSearch, Redshift, Splunk, or S3. Saving to ElasticSearch, Redshift, and Splunk would require infrastructure and VPC on us-east-1 which is not always desirable. Save to S3 is posing a difficulty because the json format makes files too large (want to save money on S3). Save to S3 on parquet format is not helping us because we don't know the format the data is in inside.
Another AWS log format that would be very useful to process with a Glue Job and query with Athena is AWS WAF logs. Today we can save it to an ElasticSearch, Redshift, Splunk, or S3. Saving to ElasticSearch, Redshift, and Splunk would require infrastructure and VPC on us-east-1 which is not always desirable. Save to S3 is posing a difficulty because the json format makes files too large (want to save money on S3). Save to S3 on parquet format is not helping us because we don't know the format the data is in inside.