search

allinurl / goaccess

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
https://goaccess.io
MIT License
18.59k stars 1.11k forks source link

GoAccess and AWS WAF access.logs (web application firewall) #2144

Open mandys opened 3 years ago

mandys commented 3 years ago

Could this be used to parse aws web application firewall logs ? Those are in Json format and in display we require requests that were blocked by firewall.

allinurl commented 3 years ago

Do you have any sample log lines that I can take a look? Thanks

mandys commented 3 years ago

This is one line in the log where the WAF has blocked a bad request.

{
    "timestamp": 1626141957554,
    "formatVersion": 1,
    "webaclId": "arn:aws:wafv2:us-west-2:912345678905:regional/webacl/WebACL/12fa5cbd-4fdc-4c3e-94cd-8e123f8fd123",
    "terminatingRuleId": "AWS-AWSManagedRulesCommonRuleSet",
    "terminatingRuleType": "MANAGED_RULE_GROUP",
    "action": "BLOCK",
    "terminatingRuleMatchDetails": [],
    "httpSourceName": "ALB",
    "httpSourceId": "912345678905-app/premium/fd1f12faadfc1d12",
    "ruleGroupList": [{
        "ruleGroupId": "AWS#AWSManagedRulesCommonRuleSet",
        "terminatingRule": {
            "ruleId": "UserAgent_BadBots_HEADER",
            "action": "BLOCK",
            "ruleMatchDetails": null
        },
        "nonTerminatingMatchingRules": [],
        "excludedRules": null
    }],
    "rateBasedRuleList": [],
    "nonTerminatingMatchingRules": [],
    "requestHeadersInserted": null,
    "responseCodeSent": null,
    "httpRequest": {
        "clientIp": "54.36.149.8",
        "country": "FR",
        "headers": [{
            "name": "Host",
            "value": "goaccess.io"
        }, {
            "name": "User-Agent",
            "value": "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
        }, {
            "name": "Accept",
            "value": "*/*"
        }, {
            "name": "Accept-Encoding",
            "value": "deflate, gzip, br"
        }],
        "uri": "/robots.txt",
        "args": "",
        "httpVersion": "HTTP/1.1",
        "httpMethod": "GET",
        "requestId": "1-60ecf505-5eba40c41bfaa68602949dd8"
    }
}
mandys commented 3 years ago

This is an allowed request

{
    "timestamp": 1626141961811,
    "formatVersion": 1,
    "webaclId": "arn:aws:wafv2:us-west-2: 912345678905:regional/webacl/WebACL/12fa5cbd-4fdc-4c3e-94cd-8e123f8fd123",
    "terminatingRuleId": "Default_Action",
    "terminatingRuleType": "REGULAR",
    "action": "ALLOW",
    "terminatingRuleMatchDetails": [],
    "httpSourceName": "ALB",
    "httpSourceId": "912345678905-app/premium/fd1f12faadfc1d12",
    "ruleGroupList": [{
        "ruleGroupId": "AWS#AWSManagedRulesCommonRuleSet",
        "terminatingRule": null,
        "nonTerminatingMatchingRules": [],
        "excludedRules": null
    }, {
        "ruleGroupId": "AWS#AWSManagedRulesSQLiRuleSet",
        "terminatingRule": null,
        "nonTerminatingMatchingRules": [],
        "excludedRules": null
    }, {
        "ruleGroupId": "AWS#AWSManagedRulesLinuxRuleSet",
        "terminatingRule": null,
        "nonTerminatingMatchingRules": [],
        "excludedRules": null
    }, {
        "ruleGroupId": "AWS#AWSManagedRulesPHPRuleSet",
        "terminatingRule": null,
        "nonTerminatingMatchingRules": [],
        "excludedRules": null
    }, {
        "ruleGroupId": "AWS#AWSManagedRulesKnownBadInputsRuleSet",
        "terminatingRule": null,
        "nonTerminatingMatchingRules": [],
        "excludedRules": null
    }],
    "rateBasedRuleList": [],
    "nonTerminatingMatchingRules": [],
    "requestHeadersInserted": null,
    "responseCodeSent": null,
    "httpRequest": {
        "clientIp": "23.239.28.171",
        "country": "US",
        "headers": [{
            "name": "Host",
            "value": "www.goaccess.io"
        }, {
            "name": "Accept",
            "value": "*/*"
        }, {
            "name": "Accept-Encoding",
            "value": "gzip"
        }, {
            "name": "User-Agent",
            "value": "Better Uptime Bot Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
        }],
        "uri": "/",
        "args": "",
        "httpVersion": "HTTP/1.1",
        "httpMethod": "GET",
        "requestId": "1-60ecf509-020da2052161001849f5658a"
    }
}
allinurl commented 3 years ago

Sorry for late reply. This should do it:

goaccess access.log --log-format='{ "timestamp": "%x", "httpRequest": { "clientIp": "%h", "headers": [{"name": "Host", "value": "%v"}, {"name":"User-Agent", "value": "%u"}], "uri": "%U", "args": "%q", "httpVersion": "%H", "httpMethod": "%m" } }' --date-format=%* --time-format=%*
allinurl commented 3 years ago

Actually you can use %e for the action, e.g., BLOCKED, ALLOWED, etc

goaccess access.log --log-format='{ "timestamp": "%x", "action": "%e", "httpRequest": { "clientIp": "%h", "headers": [{"name": "Host", "value": "%v"}, {"name":"User-Agent", "value": "%u"}], "uri": "%U", "args": "%q", "httpVersion": "%H", "httpMethod": "%m" } }' --date-format=%* --time-format=%*
mandys commented 3 years ago

Thank you for getting back. I tried

/usr/local/bin/goaccess aws-waf-log.txt goaccess access.log --log-format='{ "timestamp": "%x", "action": "%e", "httpRequest": { "clientIp": "%h", "headers": [{"name": "Host", "value": "%v"}, {"name":"User-Agent", "value": "%u"}], "uri": "%U", "args": "%q", "httpVersion": "%H", "httpMethod": "%m" } }' --date-format=% --time-format=%

Parsed 1 lines producing the following errors:
Token '6141873127,' doesn't match specifier '%x'
Format Errors - Verify your log/date/time format

Then changed timestamp to %f and tried again

/usr/local/bin/goaccess aws-waf-log.txt goaccess access.log --log-format='{ "timestamp": "%f", "action": "%e", "httpRequest": { "clientIp": "%h", "headers": [{"name": "Host", "value": "%v"}, {"name":"User-Agent", "value": "%u"}], "uri": "%U", "args": "%q", "httpVersion": "%H", "httpMethod": "%m" } }' --date-format=%* --time-format=%*
Parsed 1 lines producing the following errors:
Token 'west-2: 912345678905:regional/webacl/WebACL/60fa5cbd-4fdc-4c3e-94cd-8e540f8fd189' doesn't match specifier '%h'
Format Errors - Verify your log/date/time format

Still doesn't work.

Sample


{
  "timestamp": 1626141873127,
  "formatVersion": 1,
  "webaclId": "arn:aws:wafv2:us-west-2: 912345678905:regional/webacl/WebACL/60fa5cbd-4fdc-4c3e-94cd-8e540f8fd189",
  "terminatingRuleId": "Default_Action",
  "terminatingRuleType": "REGULAR",
  "action": "ALLOW",
  "terminatingRuleMatchDetails": [],
  "httpSourceName": "ALB",
  "httpSourceId": "912345678905-app/cname/fd6f11faadfc8d83",
  "ruleGroupList": [
    {
      "ruleGroupId": "AWS#AWSManagedRulesCommonRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    },
    {
      "ruleGroupId": "AWS#AWSManagedRulesSQLiRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    },
    {
      "ruleGroupId": "AWS#AWSManagedRulesLinuxRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    },
    {
      "ruleGroupId": "AWS#AWSManagedRulesPHPRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    },
    {
      "ruleGroupId": "AWS#AWSManagedRulesKnownBadInputsRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    }
  ],
  "rateBasedRuleList": [],
  "nonTerminatingMatchingRules": [],
  "requestHeadersInserted": null,
  "responseCodeSent": null,
  "httpRequest": {
    "clientIp": "170.178.178.163",
    "country": "US",
    "headers": [
      {
        "name": "host",
        "value": "goaccess.io"
      },
      {
        "name": "upgrade-insecure-requests",
        "value": "1"
      },
      {
        "name": "user-agent",
        "value": "Mozilla/5.0 (Linux; Android 11; SM-G9810 Build/RP1A.200720.012; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.105 Mobile Safari/537.36"
      },
      {
        "name": "accept",
        "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
      },
      {
        "name": "x-requested-with",
        "value": "net.radio"
      },
      {
        "name": "sec-fetch-site",
        "value": "none"
      },
      {
        "name": "sec-fetch-mode",
        "value": "navigate"
      },
      {
        "name": "sec-fetch-user",
        "value": "?1"
      },
      {
        "name": "sec-fetch-dest",
        "value": "document"
      },
      {
        "name": "accept-encoding",
        "value": "gzip, deflate"
      },
      {
        "name": "accept-language",
        "value": "en-GB,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,en-US;q=0.6"
      },
      {
        "name": "cookie",
        "value": "__utmc=121975063; __utmz=121975063.1596376219.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); "
      }
    ],
    "uri": "/s/landing/abcd",
    "args": "vtg_device=android&vtg_webview=1",
    "httpVersion": "HTTP/2.0",
    "httpMethod": "GET",
    "requestId": "1-71ecf4b1-0fb74c2e5e00cfc46a080aab"
  }
}
allinurl commented 3 years ago

Are you running v1.5.1 (latest)? Here's what I get for that last line you posted:

2021-07-28-081632_676x317_scrot

mandys commented 3 years ago

Just updated to 1.5.1 ( I was on 1.4 ). Code provided by you is working great!

I am able to see the block vs allow percentages which is helpful.

But, if I want to dig down a bit into which requests were blocked, would that be possible in any release ?

Currently, I don't see any waf parser and then only way is to use amazon athena, kinesis, kibana dashboard etc and launch a complex setup.

goaccess has a great opportunity to become a one stop shop.

Thanks for all the hardwork!

allinurl commented 3 years ago

I'm working on #117 as we speak, so you should be able to simply filter/search by whatever request you are interested to see and should display the relevant stats. So this is coming for sure! Also, on there is #1202 which would add much more flexibility to custom fields and panels. That would be the second biggest feature goaccess is getting.

My question to you, are there specific fields you are interested in from the JSON request you posted above?