search

pfelk / docker

Deploy pfelk with docker-compose
Apache License 2.0
56 stars 20 forks source link

can't log / logstash error #31

Closed ghost235 closed 2 years ago

ghost235 commented 3 years ago

Hi, 1st: Thanks for this great repo!!

Describe the bug I can not log into logstash (see logfiles) have I to set any file rights ?

.: insgesamt 2,1M drwxr-xr-x 6 root root 4,0K Apr 26 10:20 . drwxr-xr-x 2 root root 4,0K Apr 25 23:26 kibana drwxr-xr-x 2 root root 4,0K Apr 25 23:26 logstash drwxr-xr-x 2 root root 4,0K Apr 25 23:26 elasticsearch drwxr-xr-x 4 root root 4,0K Apr 25 23:26 etc drwxr-xr-x 5 root root 4,0K Apr 25 23:25 .. -rw-r--r-- 1 root root 62K Apr 22 22:46 pfelkdocker.zip -rw-r--r-- 1 root root 2,0M Apr 19 20:38 geoipupdate_4.7.1_linux_amd64.deb -rw-r--r-- 1 root root 2,7K Mär 26 07:57 docker-compose.yml -rw-r--r-- 1 root root 18 Mär 26 07:57 .env

./kibana: insgesamt 12K drwxr-xr-x 6 root root 4,0K Apr 26 10:20 .. drwxr-xr-x 2 root root 4,0K Apr 25 23:26 . -rw-r--r-- 1 root root 70 Mär 26 07:57 Dockerfile

./logstash: insgesamt 12K drwxr-xr-x 6 root root 4,0K Apr 26 10:20 .. drwxr-xr-x 2 root root 4,0K Apr 25 23:26 . -rw-r--r-- 1 root root 74 Mär 26 07:57 Dockerfile

./elasticsearch: insgesamt 12K drwxr-xr-x 6 root root 4,0K Apr 26 10:20 .. drwxr-xr-x 2 root root 4,0K Apr 25 23:26 . -rw-r--r-- 1 root root 84 Mär 26 07:57 Dockerfile

./etc: insgesamt 16K drwxr-xr-x 6 root root 4,0K Apr 26 10:20 .. drwxr-xr-x 5 root root 4,0K Apr 25 23:26 pfelk drwxr-xr-x 4 root root 4,0K Apr 25 23:26 . drwxr-xr-x 3 root root 4,0K Apr 25 23:26 logstash

./etc/pfelk: insgesamt 20K drwxr-xr-x 4 root root 4,0K Apr 26 09:43 conf.d drwxr-xr-x 5 root root 4,0K Apr 25 23:26 . drwxr-xr-x 2 root root 4,0K Apr 25 23:26 patterns drwxr-xr-x 4 root root 4,0K Apr 25 23:26 .. drwxr-xr-x 2 root root 4,0K Apr 25 23:26 databases

./etc/pfelk/conf.d: insgesamt 88K drwxr-xr-x 4 root root 4,0K Apr 26 09:43 . drwxr-xr-x 2 root root 4,0K Apr 26 09:01 databases drwxr-xr-x 2 root root 4,0K Apr 26 09:01 patterns drwxr-xr-x 5 root root 4,0K Apr 25 23:26 .. -rw-r--r-- 1 root root 2,0K Mär 26 07:57 01-inputs.conf -rw-r--r-- 1 root root 2,3K Mär 26 07:57 02-types.conf -rw-r--r-- 1 root root 1,2K Mär 26 07:57 03-filter.conf -rw-r--r-- 1 root root 7,2K Mär 26 07:57 05-apps.conf -rw-r--r-- 1 root root 4,1K Mär 26 07:57 20-interfaces.conf -rw-r--r-- 1 root root 4,4K Mär 26 07:57 30-geoip.conf -rw-r--r-- 1 root root 1005 Mär 26 07:57 35-rules-desc.conf -rw-r--r-- 1 root root 1,3K Mär 26 07:57 36-ports-desc.conf -rw-r--r-- 1 root root 2,1K Mär 26 07:57 37-enhanced_user_agent.conf -rw-r--r-- 1 root root 5,2K Mär 26 07:57 38-enhanced_url.conf -rw-r--r-- 1 root root 926 Mär 26 07:57 45-cleanup.conf -rw-r--r-- 1 root root 2,8K Mär 26 07:57 49-enhanced_private.conf -rw-r--r-- 1 root root 6,6K Mär 26 07:57 50-outputs.conf

./etc/pfelk/conf.d/databases: insgesamt 8,0K drwxr-xr-x 4 root root 4,0K Apr 26 09:43 .. drwxr-xr-x 2 root root 4,0K Apr 26 09:01 .

./etc/pfelk/conf.d/patterns: insgesamt 8,0K drwxr-xr-x 4 root root 4,0K Apr 26 09:43 .. drwxr-xr-x 2 root root 4,0K Apr 26 09:01 .

./etc/pfelk/patterns: insgesamt 20K drwxr-xr-x 2 root root 4,0K Apr 25 23:26 . drwxr-xr-x 5 root root 4,0K Apr 25 23:26 .. -rw-r--r-- 1 root root 9,4K Mär 26 07:57 pfelk.grok

./etc/pfelk/databases: insgesamt 132K drwxr-xr-x 5 root root 4,0K Apr 25 23:26 .. drwxr-xr-x 2 root root 4,0K Apr 25 23:26 . -rw-r--r-- 1 root root 15 Mär 26 07:57 private-hostnames.csv -rw-r--r-- 1 root root 26 Mär 26 07:57 rule-names.csv -rw-r--r-- 1 root root 116K Mär 26 07:57 service-names-port-numbers.csv

./etc/logstash: insgesamt 12K drwxr-xr-x 3 root root 4,0K Apr 25 23:26 . drwxr-xr-x 4 root root 4,0K Apr 25 23:26 .. drwxr-xr-x 2 root root 4,0K Apr 25 23:26 config

./etc/logstash/config: insgesamt 16K drwxr-xr-x 2 root root 4,0K Apr 25 23:26 . drwxr-xr-x 3 root root 4,0K Apr 25 23:26 .. -rw-r--r-- 1 root root 720 Mär 26 07:57 logstash.yml -rw-r--r-- 1 root root 893 Mär 26 07:57 pipelines.yml

To Reproduce docker-compose up

Operating System (please complete the following information):

Elasticsearch, Logstash, Kibana (please complete the following information):

a3ilson commented 3 years ago

Here is the file structure:

/etc/pfelk/
├── conf.d
│   ├── 01-inputs.conf
│   ├── 02-types.conf
│   ├── 03-filter.conf
│   ├── 05-apps.conf
│   ├── 20-interfaces.conf
│   ├── 30-geoip.conf
│   ├── 35-rules-desc.conf
│   ├── 36-ports-desc.conf
│   ├── 45-cleanup.conf
│   └── 50-outputs.conf
├── config
│   ├── logstash.yml
│   └── pipelines.yml
├── databases
│   ├── private-hostnames.csv
│   ├── rule-names.csv
│   └── service-names-port-numbers.csv
├── kibana.yml
└── patterns
    ├── openvpn.grok
    └── pfelk.grok

Your file structure looks to be good...just make sure you are using the files from this repo and this docker-compose.yml

ghost235 commented 3 years ago

I do but get this

logstash | [ERROR] 2021-04-26 12:46:44.454 [Converge PipelineAction::Create] translate - Invalid setting for translate filter plugin: logstash | logstash | filter { logstash | translate { logstash | # This setting must be a path logstash | # File does not exist or cannot be opened /etc/pfelk/databases/rule-names.csv logstash | dictionary_path => "/etc/pfelk/databases/rule-names.csv" logstash | ... logstash | } logstash | } logstash | [ERROR] 2021-04-26 12:46:44.460 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:pfelk, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:84)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:191)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:178)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]} logstash | warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true): logstash | LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<pfelk> logstash | create at org/logstash/execution/ConvergeResultExt.java:129 logstash | add at org/logstash/execution/ConvergeResultExt.java:57 logstash | converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:380 logstash | [ERROR] 2021-04-26 12:46:44.472 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<pfelk>"} logstash | [FATAL] 2021-04-26 12:46:44.483 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<pfelk>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:inadd'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:380:in `block in converge_state'"]} logstash | [FATAL] 2021-04-26 12:46:44.492 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit logstash | org.jruby.exceptions.SystemExit: (SystemExit) exit

a3ilson commented 3 years ago

try removing the 35-rules.conf file and see if it works...odds are the contents of that file are empty (not widely utilized)

ghost235 commented 3 years ago

now the error changed

logstash | [ERROR] 2021-04-27 07:00:01.114 [Converge PipelineAction::Create] translate - Invalid setting for translate filter plugin: logstash | logstash | filter { logstash | translate { logstash | # This setting must be a path logstash | # File does not exist or cannot be opened /etc/pfelk/databases/service-names-port-numbers.csv logstash | dictionary_path => "/etc/pfelk/databases/service-names-port-numbers.csv" logstash | ... logstash | } logstash | } logstash | [ERROR] 2021-04-27 07:00:01.131 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:pfelk, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:84)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:191)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:178)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:396)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:205)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]} logstash | warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true): logstash | LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<pfelk> logstash | create at org/logstash/execution/ConvergeResultExt.java:129 logstash | add at org/logstash/execution/ConvergeResultExt.java:57 logstash | converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:380 logstash | [ERROR] 2021-04-27 07:00:01.156 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<pfelk>"} logstash | [FATAL] 2021-04-27 07:00:01.185 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<pfelk>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:inadd'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:380:in `block in converge_state'"]} logstash | [FATAL] 2021-04-27 07:00:01.216 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit logstash | org.jruby.exceptions.SystemExit: (SystemExit) exit logstash | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?] logstash | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?] logstash | at usr.share.logstash.lib.bootstrap.environment.

(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

a3ilson commented 3 years ago

Seems you have something misconfigured (docker-compose.yml and/or file location.

a3ilson commented 3 years ago

What method did you utilize to install?

ghost235 commented 3 years ago

Which location it sould be?

I use https://github.com/pfelk/docker 1-4 and https://github.com/pfelk/pfelk/blob/main/install/configuration.md 1+2 skript 3,4,6,8

a3ilson commented 3 years ago

Awe...got it.

The 35-rules-desc.conf and 36-ports-desc.conf files do not currently work with this docker instance. Simple remove both files and everything should work. I renamed within the repo but need to update the Zip file.

ghost235 commented 3 years ago

May be but sees to be not the only issue. Which locationthe files sould be?

Next Error:

logstash | [ERROR] 2021-05-05 21:21:37.790 [Converge PipelineAction::Create] translate - Invalid setting for translate filter plugin: logstash | logstash | filter { logstash | translate { logstash | # This setting must be a path logstash | # File does not exist or cannot be opened /etc/pfelk/databases/private-hostnames.csv logstash | dictionary_path => "/etc/pfelk/databases/private-hostnames.csv" logstash | ... logstash | } logstash | }

a3ilson commented 3 years ago

okay...updated a few files. Retry (installing) or simply remove the 49-enhanced_private.conf file

ghost235 commented 3 years ago

OK removing 49-enhanced_private.conf do the trick to the file errors! 👍

But seems something is wron in my configuration. In under Kibana -> Stack Management -> Index patterns -> e.g. pfelk-haproxy-* -> Fields is emty

And regularly:

logstash | [ERROR] 2021-05-06 07:18:46.808 [[pfelk]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"pfelk", :exception=>#<Grok::PatternError: pattern %{HAPROXY} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1442:inloop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:288:inblock in register'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:282:inblock in register'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.3.0/lib/logstash/filters/grok.rb:277:inregister'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:inblock in register_plugins'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:inregister_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:inblock in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.conf", "/etc/pfelk/conf.d/02-types.conf", "/etc/pfelk/conf.d/03-filter.conf", "/etc/pfelk/conf.d/05-apps.conf", "/etc/pfelk/conf.d/20-interfaces.conf", "/etc/pfelk/conf.d/30-geoip.conf", "/etc/pfelk/conf.d/37-enhanced_user_agent.conf", "/etc/pfelk/conf.d/38-enhanced_url.conf", "/etc/pfelk/conf.d/45-cleanup.conf", "/etc/pfelk/conf.d/50-outputs.conf"], :thread=>"#<Thread:0x3c797eae@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}

a3ilson commented 2 years ago

@ghost235 - is this still an issue? have you tried with the latest changes?