No indexes in Kibana

[riccardospeggiorin-centropaghe]

I have installed the pfelk in docker from the zip provided and run the sh script for creating templates and dashboards. All seems ok, the port 5140 of logstash is receving packet, checked with tcpdump and saw logs from firewall ip, but the dashborad shows me an error and I cannot see any index in the kibana dashborad management

These are the logs of logstash

[INFO ] 2021-05-13 12:31:41.197 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[ERROR] 2021-05-13 12:31:41.270 [[pfelk]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"pfelk", :exception=>#<Grok::PatternError: pattern %{HAPROXY} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1442:in `loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:288:in `block in register'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:282:in `block in register'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:277:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.conf", "/etc/pfelk/conf.d/02-types.conf", "/etc/pfelk/conf.d/03-filter.conf", "/etc/pfelk/conf.d/05-apps.conf", "/etc/pfelk/conf.d/20-interfaces.conf", "/etc/pfelk/conf.d/30-geoip.conf", "/etc/pfelk/conf.d/37-enhanced_user_agent.conf", "/etc/pfelk/conf.d/38-enhanced_url.conf", "/etc/pfelk/conf.d/45-cleanup.conf", "/etc/pfelk/conf.d/50-outputs.conf"], :thread=>"#<Thread:0x78d20b07@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[INFO ] 2021-05-13 12:31:41.271 [[pfelk]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"pfelk"}
[ERROR] 2021-05-13 12:31:41.277 [Converge PipelineAction::Create<pfelk>] agent - Failed to execute action {:id=>:pfelk, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<pfelk>, action_result: false", :backtrace=>nil}
[INFO ] 2021-05-13 12:31:41.325 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-05-13 12:31:42.323 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[INFO ] 2021-05-13 12:31:43.319 [LogStash::Runner] runner - Logstash shut down.
Using bundled JDK: /usr/share/logstash/jdk

are you sure that the path are all correct? Cause in the docker-compose I see:

      - ./etc/pfelk/conf.d/patterns/:/etc/pfelk/patterns:ro
      - ./etc/pfelk/conf.d/databases/:/etc/pfelk/databases:ro

but these directories are empty. The files are in /etc/pfelk/patterns and /etc/pfelk/databases on the host

[riccardospeggiorin-centropaghe]

So after doing all from scratch and without using the zip file, all seems to work! There are some problems with the zip !

[Thrilleratplay]

@riccardospeggiorin-centropaghe Thank you for this. I spent too much time trying to figure out this issue. Can you reopen this issue as I think the zip should be fixed?

[riccardospeggiorin-centropaghe]

No problem. There are some files that are missing, like the first post

[create-atl-delete]

The docker-compose.yml in main has the correct paths. Can use wget https://raw.githubusercontent.com/pfelk/docker/main/docker-compose.yml as a workaround until .zip is fixed.