search

aws-solutions / aws-waf-security-automations

This solution automatically deploys a single web access control list (web ACL) with a set of AWS WAF rules designed to filter common web-based attacks.
https://aws.amazon.com/solutions/aws-waf-security-automations
Apache License 2.0
852 stars 366 forks source link

Glue Table for WAF Access Logs is missing some of the log fields (e.g. labels) #259

Open m-davison opened 9 months ago

m-davison commented 9 months ago

Is your feature request related to a problem? Please describe. The Glue Table created for the GlueWafAccessLogsTable does not contain the newer fields for:

These are useful fields for querying and debugging rules, especially the labels struct.

Describe the feature you'd like Add the missing fields to the GlueWafAccessLogsTable resource in the aws-waf-security-automations-firehose-athena.template so we don't need to customize the solution in order to get the extra fields when debugging rules.

Additional context See Creating a table for AWS WAF logs without partitioning for full set of fields

WillAWS commented 9 months ago

Hi @m-davison , thanks for your enhancement request. Could you provide the details of your use case and how you want to query those fields, etc.?

m-davison commented 9 months ago

The key field I'm after at the moment is labels. Whilst I don't need that it to be used by the lambdas, I do use the waf_access_logs table to debug rules I've setup or look for patterns during / after an attack to see how best to design a rule.

For example, if I have one rule setup to add a label to any web based UAs where the atp token is missing, and I want to see after a login issue which of the login requests had that label associated to it, or other labels e.g. standard ATP or BOT labels that I could potentially use.

Plus I think It would just be good to have access to all the data in the WAF access logs, rather than just a subset so I don't need to either customise the solution or create a second table for debugging purposes.

WillAWS commented 9 months ago

@m-davison Currently this is not in the scope of the out-of-box solution as the main purpose of the log analysis feature is to automate detecting and blocking common attacks via a built-in Lambda or Athena log parser using necessary fields instead of bringing the whole data. We can add your request to our backlog, but in general if you have specific needs like this, we recommend you to customize the source code to get fast resolution.

m-davison commented 9 months ago

No worries... Thanks @WillAWS ... I have customised it, just thought I'd raise it in case it could be of use to others too