lephisto / pfsense-analytics

Pfsense Analytics w/ Graylog, Elasticsearch, InfluxDB and Grafana fully dockerized for Firewall and DPI..
GNU General Public License v3.0
156 stars 56 forks source link

Content pack issue, Graylog v3.3.4 #40

Closed wintrmte closed 3 years ago

wintrmte commented 4 years ago

Getting an error uploading the Content Pack on Graylog 3.3.4 -- from the log file:

Caused by: org.graylog2.contentpacks.exceptions.DivergingEntityConfigurationException: Expected Grok pattern for name "COMMONAPACHELOG": <%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)>; actual Grok pattern: <%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} [%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:mm:ss Z}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)>

Looks like it has to do with this entry in the content pack:

{ "v": "1", "type": { "name": "grok_pattern", "version": "1" }, "id": "7482a5f4-868c-4ef2-839f-a22141445c5c", "data": { "name": "COMMONAPACHELOG", "pattern": "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" }, "constraints": [ { "type": "server-version", "version": ">=3.1.2+9e96b08" } ] },

Removing this will allow the content pack to install, but breaks some of the output.

Not sure why this is happening.. Any ideas?

kuestess commented 4 years ago

@wintrmte I had the same issue - it looks like this pattern already exists in Graylog. If you go to System->Grok patterns and delete the two Apache patterns and then import/install the content pack, it works just fine.

lephisto commented 3 years ago

I guess I will need to dig into it with newer Versions. Maybe in the next few weeks ill take the time to update all this.

lephisto commented 3 years ago

You might want to check branch gl4_es7, this is basic work for going to Graylog4 and Elasticsearch 7. I didn't bother with GL3..