search

fluent / fluentd-docs

This repository is deprecated. Go to fluentd-docs-gitbook repository.
49 stars 119 forks source link

Need fluentd parsing for NGINX NAXSI Logs #605

Open aarvee11 opened 5 years ago

aarvee11 commented 5 years ago

Hi All,

We need to parse NAXSI_FMT and NAXSI_EXLOG logs to be parsed by fluentd which shall be further shipped into elasticsearch. We do not want to build logstash for just parsing the logs using GORK patterns instead use fluentd's excellent capability in performing the same.

A sample log is given below:

2019/01/16 07:34:06 [error] 46#0: *17 NAXSI_FMT: ip=172.17.0.1&server=api.secops.in&uri=/v1/collect/pay_BkdDYXLVTVzuG10436/callback/3cbca6c56156bcad278025c450e7551b1e467bd9/secops_live_ILgsfZjhaskjfhZoFIKMb&vers=0.56&total_processed=3&total_blocked=3&config=learning&cscore0=$SQL&score0=492&cscore1=$XSS&score1=472&zone0=HEADERS&id0=1001&var_name0=cookie&zone1=HEADERS&id1=1010&var_name1=cookie&zone2=HEADERS&id2=1015&var_name2=cookie&zone3=HEADERS&id3=1310&var_name3=cookie&zone4=HEADERS&id4=1311&var_name4=cookie&zone5=HEADERS&id5=1315&var_name5=cookie&zone6=BODY&id6=1002&var_name6=pares&zone7=BODY&id7=1009&var_name7=pares, client: 172.17.0.1, server: , request: "POST /v1/collect/pay_BkdDYXLVTVzuG10436/callback/3cbca6c56156bcad278025c450e7551b1e467bd9/rzp_live_ILgsfZCZoFIKMb HTTP/1.1", host: "api.secops.in", referrer: "https://xxx.yyyyyyyy.com/xxx-web-v2?ISSUER_ID=8117"

For further reading please refer to NAXSI_Logs_Documentation

cosmo0920 commented 5 years ago

@okkez Could you take a look? I guess that we can handle this log with fluent-plugin-grok-parser .

okkez commented 5 years ago

@aarvee11 You can use grok patterns with fluent-plugin-grok-parser if you want to parse your logs using grok patterns. But built-in grok patterns cannot parse NAXSI_FMT and NAXSI_EXLOG logs. You can add custom grok patterns to parse your logs, see fluent-plugin-grok-parser's document for more details.

Or you can use built-in parser_regexp plugin.