search

enotspe / fortinet-2-elasticsearch

Fortinet products logs to Elasticsearch
Apache License 2.0
86 stars 39 forks source link

Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable #55

Closed stroesa closed 9 months ago

stroesa commented 11 months ago

Running into an issue with my setup. Some background:

Installed everything as per instructions but getting this error:

Nov 14 13:32:17 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:17,417][INFO ][logstash.javapipeline ] Pipelinesyslog-fortinet-common_ecs-outputis configured withpipeline.ecs_compatibility: v8setting. All plugins in this pipeline will default toecs_compatibility => v8unless explicitly configured otherwise. Nov 14 13:32:17 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:17,515][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate-input5424-kv][6aab6188921cec0832a0712bc324ef942bf88174229dcfed0e9b06c29785d59a] Attempted to send event to 'syslog-fortinet-fortigate_2_ecsv2' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 14 13:32:17 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:17,555][INFO ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.17.7.134:9200"]} Nov 14 13:32:17 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:17,573][WARN ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure setssl_verification_mode => full Nov 14 13:32:18 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:18,125][INFO ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@172.17.7.134:9200/]}} Nov 14 13:32:18 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:18,514][INFO ][logstash.javapipeline ][syslog-fortinet-fortigate_2_ecsv2] Pipeline Java execution initialization time {"seconds"=>2.38} Nov 14 13:32:18 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:18,516][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate-input5424-kv][6aab6188921cec0832a0712bc324ef942bf88174229dcfed0e9b06c29785d59a] Attempted to send event to 'syslog-fortinet-fortigate_2_ecsv2' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 14 13:32:18 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:18,524][INFO ][logstash.javapipeline ][syslog-fortinet-fortigate_2_ecsv2] Pipeline started {"pipeline.id"=>"syslog-fortinet-fortigate_2_ecsv2"} Nov 14 13:32:18 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:18,825][WARN ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@172.17.7.134:9200/"} Nov 14 13:32:18 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:18,827][INFO ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Elasticsearch version determined (8.10.4) {:es_version=>8} Nov 14 13:32:18 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:18,828][WARN ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Detected a 6.x and above cluster: thetypeevent field won't be used to determine the document _type {:es_version=>8} Nov 14 13:32:18 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:18,843][WARN ][logstash.filters.grok ][syslog-fortinet-common_ecs-output] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated Nov 14 13:32:19 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:19,807][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 14 13:32:20 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:20,031][INFO ][logstash.filters.geoip.downloadmanager] new database version detected? true Nov 14 13:32:20 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:20,821][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 14 13:32:21 zavpemblogs31 logstash[6565]: [2023-11-14T13:32:21,822][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

stroesa commented 11 months ago

I have uninstalled logstash-input-twitter and installed logstash-filter-tld version 3.1.3 but still gettting the same error

stroesa commented 11 months ago

Nov 16 09:53:47 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:47,277][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:48 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:48,278][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:49 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:49,278][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:50 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:50,278][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:51 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:51,279][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:52 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:52,279][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:53 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:53,280][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:54 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:54,280][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:55 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:55,280][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:56 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:56,281][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:57 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:57,281][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:58 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:58,282][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:59 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:59,282][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:54:00 zavpemblogs31 logstash[1948]: [2023-11-16T09:54:00,283][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:54:01 zavpemblogs31 logstash[1948]: [2023-11-16T09:54:01,283][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

enotspe commented 11 months ago

Those logs are not really helpfull.

I need the logs when logstash starts. Normally what comes before/next "running pipelines/not running pipelines"

In my case, that lines is:

[2023-11-20T15:49:28,392][INFO ][logstash.agent ] Pipelines running {:count=>17, :running_pipelines=>[:"syslog-fortinet-fortiedr_2_ecs-default", :"syslog-fortinet-fortigate-input-kv-client1", :"syslog-fortinet-fortimail-input-kv-client2", :"syslog-fortinet-forticlient_2_ecs-client2", :"syslog-fortinet-forticlient-input-kv-client2", :"syslog-fortinet-fortimail_2_ecs-client2", :"syslog-fortinet-fortigate-input-kv-client35424", :"syslog-fortinet-fortiedr-input-kv-default", :"syslog-fortinet-fortigate-input-kv-client15424", :"syslog-fortinet-fortigate-input-kv-client25424", :"syslog-fortinet-fortigate_2_ecs-client1", :"syslog-fortinet-fortigate_2_ecs-client3", :"syslog-fortinet-fortigate_2_ecs-client2", :"syslog-fortinet-common_ecs-output-default", :"syslog-fortinet-common_ecs-output-client3", :"syslog-fortinet-common_ecs-output-client1", :"syslog-fortinet-common_ecs-output-client2"], :non_running_pipelines=>[]}

GoingCrazy90 commented 10 months ago

I get the same error: [2023-12-09T20:38:02,243][INFO ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@172.31.0.122:9200/]}} [2023-12-09T20:38:02,459][WARN ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@172.31.0.122:9200/"} [2023-12-09T20:38:02,460][INFO ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Elasticsearch version determined (8.11.1) {:es_version=>8} [2023-12-09T20:38:02,460][WARN ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8} [2023-12-09T20:38:02,475][WARN ][logstash.filters.grok ][syslog-fortinet-common_ecs-output] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated [2023-12-09T20:38:02,520][WARN ][logstash.filters.grok ][syslog-fortinet-common_ecs-output] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated [2023-12-09T20:38:02,540][ERROR][logstash.javapipeline ][syslog-fortinet-common_ecs-output] Pipeline error {:pipeline_id=>"syslog-fortinet-common_ecs-output", :exception=>#<ArgumentError: wrong number of arguments (given 2, expected 1)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/public_suffix-3.1.1/lib/public_suffix/list.rb:69:in parse'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/public_suffix-3.1.1/lib/public_suffix/list.rb:51:indefault'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-tld-3.1.2/lib/logstash/filters/tld.rb:33:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in block in register_plugins'", "org/jruby/RubyArray.java:1987:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:611:inmaybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:249:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/syslog-fortinet-common_ecs-output.conf"], :thread=>"#<Thread:0x263d4cf6 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"} [2023-12-09T20:38:02,541][INFO ][logstash.javapipeline ][syslog-fortinet-common_ecs-output] Pipeline terminated {"pipeline.id"=>"syslog-fortinet-common_ecs-output"} [2023-12-09T20:38:02,551][ERROR][logstash.agent ] Failed to execute action {:id=>:"syslog-fortinet-common_ecs-output", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil} [2023-12-09T20:38:03,067][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. [2023-12-09T20:38:04,068][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. [2023-12-09T20:38:05,068][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

As soon as i uncomment the TLD Lines in Output all works.

enotspe commented 10 months ago

it does not seem you are using logstash-filter-tld --version 3.1.3

[2023-12-09T20:38:02,540][ERROR][logstash.javapipeline ][syslog-fortinet-common_ecs-output] Pipeline error {:pipeline_id=>"syslog-fortinet-common_ecs-output", :exception=>#<ArgumentError: wrong number of arguments (given 2, expected 1)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/public_suffix-3.1.1/lib/public_suffix/list.rb:69:in parse'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/public_suffix-3.1.1/lib/public_suffix/list.rb:51:in default'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-tld-3.1.2/lib/logstash/filters/tld.rb:33:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in block in register_plugins'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:611:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:249:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/syslog-fortinet-common_ecs-output.conf"], :thread=>"#<Thread:0x263d4cf6 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}

after these steps

Running Logstash 8.10 Elasticsearch 8.10 Removed logstash-input-twitter Installed logstash-filter-tld --version 3.1.3

please restart logstash. Probably you already did, but just want to make sure.

For some reason your plugin is not loading on version 3.1.3

GoingCrazy90 commented 10 months ago

Had to update the tld Plugin manually to version 3.1.3 (this command gave me 3.1.2: bin/logstash-plugin install logstash-filter-tld)

Now it works fine. My issue is resolved. Thanks very much for your support :)

By the way: very nice Solution!