search

pfelk / docker

Deploy pfelk with docker-compose
Apache License 2.0
57 stars 20 forks source link

Logstash Grok::PatternError: pattern %{SNORT} not defined #7

Closed SaymonDzen closed 4 years ago

SaymonDzen commented 4 years ago

Describe the bug Data does not flow to elasticksearch.

To Reproduce Configured by default. Changed only ip Pfsence and maxmind added in docker. In logs logstash See error [ERROR] 2020-06-10 08:57:49.898 [[main]-pipeline-manager] javapipeline - Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{SNORT} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1442:inloop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:288:inblock in register'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:282:inblock in register'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:277:inregister'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:216:inblock in register_plugins'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:215:inregister_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:521:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125:inblock in start'"], "pipeline.sources"=>["/usr/share/logstash/etc/logstash/conf.d/01-inputs.conf", "/usr/share/logstash/etc/logstash/conf.d/05-firewall.conf", "/usr/share/logstash/etc/logstash/conf.d/10-others.conf", "/usr/share/logstash/etc/logstash/conf.d/20-suricata.conf", "/usr/share/logstash/etc/logstash/conf.d/25-snort.conf", "/usr/share/logstash/etc/logstash/conf.d/30-geoip.conf", "/usr/share/logstash/etc/logstash/conf.d/40-dns.conf", "/usr/share/logstash/etc/logstash/conf.d/45-cleanup.conf", "/usr/share/logstash/etc/logstash/conf.d/50-outputs.conf"], :thread=>"#<Thread:0x66ea3b06@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:121 run>"} [ERROR] 2020-06-10 08:57:49.913 [Converge PipelineAction::Create

] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create
, action_result: false", :backtrace=>nil}

Screenshots kibana https://www.dropbox.com/s/xwl1x9578mfdpfo/%D0%A1%D0%BD%D0%B8%D0%BC%D0%BE%D0%BA.PNG?dl=0

Operating System (please complete the following information):

  • OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): Linux 4.19.0-9-amd64 x86_64 PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

  • Version of Docker (docker -v): Docker version 19.03.11, build 42e35e61f3

  • Version of Docker-Compose (docker-compose -v): docker-compose version 1.26.0, build unknown Elasticsearch, Logstash, Kibana (please complete the following information):

  • Version of ELK (cat /docker-pfelk/.env) ELK_VERSION=7.7.0 **Service logs

  • docker-compose logs pfelk01

  • docker-compose logs pfelk02

  • docker-compose logs pfelk03

  • docker-compose logs logstash

  • docker-compose logs kibana https://www.dropbox.com/s/r4knplbwaxkwi6e/logs.zip?dl=0 Additional context Add any other context about the problem here.

a3ilson commented 4 years ago

@SaymonDzen - thanks and I’ll test and troubleshoot this within the week.

Initially, it appears there may be an issue with the setup more so an issue communicating with elastic. The error indicating the referenced snort pattern is missing and likely a missing or misplace file location of the grok pattern.

SaymonDzen commented 4 years ago

i removing the 25-snort.conf and restart logstash. the problem resolved and the data went to the elastic.

a3ilson commented 4 years ago

@SaymonDzen - Did you download the Zip file or did you manually download the corresponding files? I noted a missing "/" from the snort file on line 8. I just corrected the omission but did not note it within the contained Zip file.

If you downloaded from the file independent (not the ZIp). The issue is corrected...otherwise I'll troubleshoot later this week.

Thanks!

SaymonDzen commented 4 years ago

this issue resolved!