Access log to Elasticsearch stop working

underscoredje commented 3 years ago

Hi!

I made a fresh install of a new vulture cluster on freebsd12.1 and connected a ES repo for logs (as I I did for my older Vulture cluster).

On this new cluster:

Pf logs are send to ES, but access logs are not. (On ES vulture_pf-* index is created, but vulture_access-* is not).
I did some checks /usr/local/etc/rsyslog.d/rsyslogd.conf file and did some tests with the part concerning my access log file:
```
input(type="imfile" File="/var/log/Vulture/worker/vulture-test.test.com--access.log" Tag="5f8d5064658ec17905c6c1b0")
```

if $syslogtag contains '5f8d5064658ec17905c6c1b0' then { action(type="mmjsonparse") $template srchidx_5f8d5064658ec17905c6c1b0,"vulture_access-%timegenerated:::date-year%-%timegenerated:::date-month%-%timegenerated:::date-day%"

Send logs to Elastic

if $parsesuccess == "OK" then { if $!score == "-" then { set $!score = 0; } else { set $!score = cnum($!score); } if $!threshold == "-" then { set $!threshold = 0; } else { set $!threshold = cnum($!threshold); }

  action(type="omelasticsearch"
       server=["http://es.test.lan:9200"]
       template="tpl_elastic"
       searchIndex="srchidx_5f8d5064658ec17905c6c1b0"
       dynSearchIndex="on"
       searchType="vulture_access"

       bulkmode="on"
       maxbytes="100m"
       queue.type="linkedlist"
       queue.size="5000"
       queue.dequeuebatchsize="300"
       action.resumeretrycount="-1")

} stop }



If I comment `if $parsesuccess == "OK" then {` line, and the correponding `}` line, `vulture-access-*` is created in ES. (but with wrong parsing).

So, i guess there is a problem with the parsing made by mmjsonparse.
I've tested with rsyslogd 8.1911.0 and 8.2008.0.

On older clusters (where it is working), rsyslogd 8.40.0 is used.

Any idea for correcting this ?

underscoredje commented 3 years ago

After more investigations, it seems the the problem is in the access_log formatting:

@cee:{"app_name":"test","src_ip":"****","user":"-","time":"2020-10-19T14:02:09+0000","http_method":"GET","requested_uri":"/,"http_code":304,"incoming_protocol":"HTTP/1.1","referer":"https://test.test.com/","user_agent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0","size":0,"bytes_received":702,"bytes_sent":177,"time_elapsed":2094,"country":"FR","city":"-","lat":"-","lon":"-","reputation":"-,-,-,-,-","owasp_top10":"-","reasons":"[]","threshold":"18","score":"0"}"authentication":"-","authentication_result":"-"}

Note the appending "authentication":"-","authentication_result":"-"}

underscoredje commented 3 years ago

OK, found the root cause and corrected it by modifying /home/vlt-gui/vulture/gui/models/modlog_settings.py. I will submit a patch.

underscoredje commented 3 years ago

PR submitted #12

b4b857f6ee commented 3 years ago

@underscoredje Brooo help me too :D. I got the vulture_pf working on the ELK. i have to modify the file like your first post?

b4b857f6ee commented 3 years ago

Oh ok this is guess :

underscoredje commented 3 years ago

Yes! Hope it is working for you now.

b4b857f6ee commented 3 years ago

@underscoredje Yes, i'm starting the Vulture Grafana dashboard to night.

KGuillemot commented 3 years ago

Hello @underscoredje, PR #12 has been merged, can you confirm me your problem is solved please ?

VultureProject / vulture3-gui

Access log to Elasticsearch stop working #11

Send logs to Elastic