search

robcowart / synesis_lite_suricata

Suricata IDS/IPS log analytics using the Elastic Stack.
Other
233 stars 92 forks source link

[Re]Some problem inside my Elasticstack + Suricata #32

Closed ainayves closed 3 years ago

ainayves commented 3 years ago

I solved my last problem, but now, different problems appear in the logs of Elastic and Kibana

Elastic log error

` aina@elasticsearch:~$ sudo head /var/log/elasticsearch/elasticsearch.log [2021-07-28T13:22:52,898][WARN ][o.e.x.m.e.l.LocalExporter] [eY6v6GM] unexpected error while indexing monitoring document org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-6.8.0.jar:6.8.0] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?] at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]

**Kibana log error**

Jul 28 15:44:48 kibana kibana[1530]: {"type":"log","@timestamp":"2021-07-28T12:44:48Z","tags":["error","task_manager"],"pid":1530,"message":"Failed to poll for work: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_doc/Maps-maps_telemetry/_update\",\"query\":{\"if_seq_no\":11,\"if_primary_term\":2,\"refresh\":\"true\"},\"body\":\"{\\"doc\\":{\\"type\\":\\"task\\",\\"task\\":{\\"taskType\\":\\"maps_telemetry\\",\\"state\\":\\"{\\\\"runs\\\\":1,\\\\"stats\\\\":{\\\\"mapsTotalCount\\\\":0,\\\\"timeCaptured\\\\":\\\\"2021-07-12T10:00:18.993Z\\\\",\\\\"attributesPerMap\\\\":{\\\\"dataSourcesCount\\\\":{\\\\"min\\\\":0,\\\\"max\\\\":0,\\\\"avg\\\\":0},\\\\"layersCount\\\\":{\\\\"min\\\\":0,\\\\"max\\\\":0,\\\\"avg\\\\":0},\\\\"layerTypesCount\\\\":{},\\\\"emsVectorLayersCount\\\\":{}}}}\\",\\"params\\":\\"{}\\",\\"attempts\\":0,\\"scheduledAt\\":\\"2021-07-12T10:00:14.897Z\\",\\"runAt\\":\\"2021-07-28T12:45:48.369Z\\",\\"status\\":\\"running\\"},\\"kibana\\":{\\"uuid\\":\\"c9ffff37-0cdd-43c4-b95c-ca38ea93aee8\\",\\"version\\":6080399,\\"apiVersion\\":1}}}\",\"statusCode\":403,\"response\":\"{\\"error\\":{\\"root_cause\\":[{\\"type\\":\\"cluster_block_exception\\",\\"reason\\":\\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\"}],\\"type\\":\\"cluster_block_exception\\",\\"reason\\":\\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\"},\\"status\\":403}\"}"}

` Logstash seems to be working :

` Jul 28 14:49:20 logstash logstash[2264]: [2021-07-28T14:49:20,449][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/synlite_suricata/geoipdbs/GeoLite2-City.mmdb"} Jul 28 14:49:20 logstash logstash[2264]: [2021-07-28T14:49:20,767][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/synlite-suricata_stats-1.0.1 Jul 28 14:49:20 logstash logstash[2264]: [2021-07-28T14:49:20,769][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/synlite-suricata-1.0.1 Jul 28 14:49:20 logstash logstash[2264]: [2021-07-28T14:49:20,882][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/synlite_suricata/geoipdbs/GeoLite2-ASN.mmdb"}

`

And suricata is working well with fielbeat

I am using ELK 6.8.0

Could you help me please???

ainayves commented 3 years ago

My elasticsearch.yml

network.host: 0.0.0.0 http.port: 9200 indices.query.bool.max_clause_count: 8192 search.max_buckets: 250000

My kibana.yml

server.port: 5601 server.host: "192.168.56.108" elasticsearch.hosts: ["http://192.168.56.103:9200"]

ainayves commented 3 years ago

The error on Kibana GUI Capture

robcowart commented 3 years ago

Can you send the Filebeat logs?

ainayves commented 3 years ago

Can you send the Filebeat logs?

Filebeat logs

` 2021-07-28T16:47:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":1}},"total":{"ticks":189750,"time":{"ms":6},"value":189750},"user":{"ticks":181560,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66330116}},"memstats":{"gc_next":160857440,"memory_alloc":81089160,"memory_total":937463480,"rss":-163840}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.08,"15":1.18,"5":1.18,"norm":{"1":1.08,"15":1.18,"5":1.18}}}}}}

`

robcowart commented 3 years ago

Is that the entire log? It doesn't look like it.

ainayves commented 3 years ago 
2021-07-28T16:35:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8130},"total":{"ticks":187830,"time":{"ms":8},"value":187830},"user":{"ticks":179700,"time":{"ms":8}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65580115}},"memstats":{"gc_next":160860992,"memory_alloc":80998232,"memory_total":930295944}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.1,"15":1.19,"5":1.2,"norm":{"1":1.1,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:35:49.590+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:35:49.590+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1455 reconnect attempt(s)
2021-07-28T16:35:49.591+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:35:49.591+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:35:49.591+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:35:49.591+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:35:58.408+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8130,"time":{"ms":1}},"total":{"ticks":187840,"time":{"ms":6},"value":187840},"user":{"ticks":179710,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65610118}},"memstats":{"gc_next":160860992,"memory_alloc":81445208,"memory_total":930742920}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.16,"15":1.2,"5":1.21,"norm":{"1":1.16,"15":1.2,"5":1.21}}}}}}
2021-07-28T16:36:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8140,"time":{"ms":6}},"total":{"ticks":188150,"time":{"ms":304},"value":188150},"user":{"ticks":180010,"time":{"ms":298}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65640117}},"memstats":{"gc_next":160857440,"memory_alloc":80590152,"memory_total":931018080}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.11,"15":1.19,"5":1.19,"norm":{"1":1.11,"15":1.19,"5":1.19}}}}}}
2021-07-28T16:36:38.481+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:36:38.481+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1456 reconnect attempt(s)
2021-07-28T16:36:38.481+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:36:38.481+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:36:38.481+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:36:38.481+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:36:58.408+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8140,"time":{"ms":5}},"total":{"ticks":188150,"time":{"ms":8},"value":188150},"user":{"ticks":180010,"time":{"ms":3}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65670115}},"memstats":{"gc_next":160857440,"memory_alloc":80719032,"memory_total":931146960}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.33,"15":1.21,"5":1.24,"norm":{"1":1.33,"15":1.21,"5":1.24}}}}}}
2021-07-28T16:37:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8140,"time":{"ms":3}},"total":{"ticks":188150,"time":{"ms":6},"value":188150},"user":{"ticks":180010,"time":{"ms":3}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65700115}},"memstats":{"gc_next":160857440,"memory_alloc":81005320,"memory_total":931433248}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.88,"15":1.17,"5":1.13,"norm":{"1":0.88,"15":1.17,"5":1.13}}}}}}
2021-07-28T16:37:35.711+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:37:35.711+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1457 reconnect attempt(s)
2021-07-28T16:37:35.712+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:37:35.712+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:37:35.712+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:37:35.712+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:37:58.409+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8150,"time":{"ms":1}},"total":{"ticks":188170,"time":{"ms":8},"value":188170},"user":{"ticks":180020,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65730115}},"memstats":{"gc_next":160857440,"memory_alloc":81288824,"memory_total":931716752}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.84,"15":1.16,"5":1.1,"norm":{"1":0.84,"15":1.16,"5":1.1}}}}}}
2021-07-28T16:38:26.127+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:38:26.127+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1458 reconnect attempt(s)
2021-07-28T16:38:26.127+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:38:26.128+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:38:26.128+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:38:26.128+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:38:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8150},"total":{"ticks":188480,"time":{"ms":307},"value":188480},"user":{"ticks":180330,"time":{"ms":307}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65760115}},"memstats":{"gc_next":160858432,"memory_alloc":80435952,"memory_total":931998384}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.19,"15":1.18,"5":1.16,"norm":{"1":1.19,"15":1.18,"5":1.16}}}}}}
2021-07-28T16:38:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8150},"total":{"ticks":188480,"time":{"ms":5},"value":188480},"user":{"ticks":180330,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65790115}},"memstats":{"gc_next":160858432,"memory_alloc":80720696,"memory_total":932283128}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.41,"15":1.2,"5":1.21,"norm":{"1":1.41,"15":1.2,"5":1.21}}}}}}
2021-07-28T16:39:14.245+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:39:14.245+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1459 reconnect attempt(s)
2021-07-28T16:39:14.245+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:39:14.246+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:39:14.246+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:39:14.246+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:39:28.409+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8150,"time":{"ms":8}},"total":{"ticks":188480,"time":{"ms":8},"value":188480},"user":{"ticks":180330}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65820115}},"memstats":{"gc_next":160858432,"memory_alloc":81004616,"memory_total":932567048}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.4,"15":1.2,"5":1.22,"norm":{"1":1.4,"15":1.2,"5":1.22}}}}}}
2021-07-28T16:39:58.408+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8160,"time":{"ms":4}},"total":{"ticks":188500,"time":{"ms":6},"value":188500},"user":{"ticks":180340,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65850115}},"memstats":{"gc_next":160858432,"memory_alloc":81282088,"memory_total":932844520}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.23,"15":1.19,"5":1.2,"norm":{"1":1.23,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:40:13.300+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:40:13.300+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1460 reconnect attempt(s)
2021-07-28T16:40:13.300+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:40:13.300+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:40:13.301+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:40:13.301+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:40:28.607+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170,"time":{"ms":11}},"total":{"ticks":188780,"time":{"ms":289},"value":188780},"user":{"ticks":180610,"time":{"ms":278}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65880318}},"memstats":{"gc_next":160871744,"memory_alloc":81566072,"memory_total":933128504}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.25,"15":1.19,"5":1.2,"norm":{"1":1.25,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:40:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170,"time":{"ms":3}},"total":{"ticks":188800,"time":{"ms":16},"value":188800},"user":{"ticks":180630,"time":{"ms":13}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65910116}},"memstats":{"gc_next":160871744,"memory_alloc":80866888,"memory_total":933560080}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.19,"15":1.18,"5":1.18,"norm":{"1":1.19,"15":1.18,"5":1.18}}}}}}
2021-07-28T16:41:02.569+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:41:02.569+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1461 reconnect attempt(s)
2021-07-28T16:41:02.570+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:41:02.570+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:41:02.570+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:41:02.570+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:41:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170,"time":{"ms":1}},"total":{"ticks":188800,"time":{"ms":8},"value":188800},"user":{"ticks":180630,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65940115}},"memstats":{"gc_next":160871744,"memory_alloc":80996824,"memory_total":933690016}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.18,"15":1.18,"5":1.18,"norm":{"1":1.18,"15":1.18,"5":1.18}}}}}}
2021-07-28T16:41:43.008+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:41:43.009+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1462 reconnect attempt(s)
2021-07-28T16:41:43.009+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:41:43.009+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:41:43.009+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:41:43.009+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:41:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170},"total":{"ticks":188810,"time":{"ms":6},"value":188810},"user":{"ticks":180640,"time":{"ms":6}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65970118}},"memstats":{"gc_next":160871744,"memory_alloc":81280360,"memory_total":933973552}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.29,"15":1.19,"5":1.21,"norm":{"1":1.29,"15":1.19,"5":1.21}}}}}}
2021-07-28T16:42:22.864+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:42:22.864+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1463 reconnect attempt(s)
2021-07-28T16:42:22.864+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:42:22.864+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:42:22.864+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:42:22.864+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:42:28.408+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170,"time":{"ms":1}},"total":{"ticks":188820,"time":{"ms":7},"value":188820},"user":{"ticks":180650,"time":{"ms":6}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66000116}},"memstats":{"gc_next":160871744,"memory_alloc":81572360,"memory_total":934265552,"rss":270336}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.47,"15":1.21,"5":1.26,"norm":{"1":1.47,"15":1.21,"5":1.26}}}}}}
2021-07-28T16:42:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180,"time":{"ms":2}},"total":{"ticks":189130,"time":{"ms":303},"value":189130},"user":{"ticks":180950,"time":{"ms":301}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66030116}},"memstats":{"gc_next":160860960,"memory_alloc":80523288,"memory_total":934622168}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.23,"15":1.2,"5":1.23,"norm":{"1":1.23,"15":1.2,"5":1.23}}}}}}
2021-07-28T16:42:58.535+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:42:58.535+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1464 reconnect attempt(s)
2021-07-28T16:42:58.535+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:42:58.535+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:42:58.535+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:42:58.535+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:43:28.408+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180},"total":{"ticks":189130,"time":{"ms":7},"value":189130},"user":{"ticks":180950,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66060115}},"memstats":{"gc_next":160860960,"memory_alloc":80734808,"memory_total":934833688}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.18,"15":1.2,"5":1.22,"norm":{"1":1.18,"15":1.2,"5":1.22}}}}}}
2021-07-28T16:43:40.284+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:43:40.285+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1465 reconnect attempt(s)
2021-07-28T16:43:40.285+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:43:40.285+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:43:40.285+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:43:40.285+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:43:58.412+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180,"time":{"ms":1}},"total":{"ticks":189140,"time":{"ms":8},"value":189140},"user":{"ticks":180960,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66090119}},"memstats":{"gc_next":160860960,"memory_alloc":81018376,"memory_total":935117256}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.84,"15":1.17,"5":1.14,"norm":{"1":0.84,"15":1.17,"5":1.14}}}}}}
2021-07-28T16:44:13.279+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:44:13.279+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1466 reconnect attempt(s)
2021-07-28T16:44:13.280+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:44:13.280+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:44:13.280+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:44:13.280+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:44:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180},"total":{"ticks":189150,"time":{"ms":7},"value":189150},"user":{"ticks":180970,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66120116}},"memstats":{"gc_next":160860960,"memory_alloc":81378696,"memory_total":935477576}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.04,"15":1.18,"5":1.16,"norm":{"1":1.04,"15":1.18,"5":1.16}}}}}}
2021-07-28T16:44:54.692+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:44:54.692+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1467 reconnect attempt(s)
2021-07-28T16:44:54.692+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:44:54.692+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:44:54.692+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:44:54.692+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:44:58.408+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180,"time":{"ms":9}},"total":{"ticks":189420,"time":{"ms":283},"value":189420},"user":{"ticks":181240,"time":{"ms":274}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66150117}},"memstats":{"gc_next":160857440,"memory_alloc":80528296,"memory_total":935765232}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.28,"15":1.19,"5":1.21,"norm":{"1":1.28,"15":1.19,"5":1.21}}}}}}
2021-07-28T16:45:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":2}},"total":{"ticks":189430,"time":{"ms":4},"value":189430},"user":{"ticks":181240,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66180118}},"memstats":{"gc_next":160857440,"memory_alloc":80731704,"memory_total":935968640}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.27,"15":1.2,"5":1.22,"norm":{"1":1.27,"15":1.2,"5":1.22}}}}}}
2021-07-28T16:45:33.707+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:45:33.707+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1468 reconnect attempt(s)
2021-07-28T16:45:33.708+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:45:33.708+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:45:33.708+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:45:33.708+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:45:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":1}},"total":{"ticks":189440,"time":{"ms":7},"value":189440},"user":{"ticks":181250,"time":{"ms":6}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66210115}},"memstats":{"gc_next":160857440,"memory_alloc":81013480,"memory_total":936250416}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.02,"15":1.18,"5":1.16,"norm":{"1":1.02,"15":1.18,"5":1.16}}}}}}
2021-07-28T16:46:22.414+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:46:22.415+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1469 reconnect attempt(s)
2021-07-28T16:46:22.416+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:46:22.416+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:46:22.416+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:46:22.416+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:46:28.411+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190},"total":{"ticks":189450,"time":{"ms":7},"value":189450},"user":{"ticks":181260,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66240117}},"memstats":{"gc_next":160857440,"memory_alloc":81300088,"memory_total":936537024}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.23,"15":1.19,"5":1.2,"norm":{"1":1.23,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:46:58.408+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190},"total":{"ticks":189740,"time":{"ms":295},"value":189740},"user":{"ticks":181550,"time":{"ms":295}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66270115}},"memstats":{"gc_next":160857440,"memory_alloc":80436368,"memory_total":936810688}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.34,"15":1.2,"5":1.23,"norm":{"1":1.34,"15":1.2,"5":1.23}}}}}}
2021-07-28T16:47:14.244+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:47:14.244+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1470 reconnect attempt(s)
2021-07-28T16:47:14.244+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:47:14.244+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:47:14.244+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:47:14.244+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:47:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":4}},"total":{"ticks":189740,"time":{"ms":8},"value":189740},"user":{"ticks":181550,"time":{"ms":4}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66300118}},"memstats":{"gc_next":160857440,"memory_alloc":80728616,"memory_total":937102936}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.34,"15":1.2,"5":1.24,"norm":{"1":1.34,"15":1.2,"5":1.24}}}}}}
2021-07-28T16:47:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":1}},"total":{"ticks":189750,"time":{"ms":6},"value":189750},"user":{"ticks":181560,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66330116}},"memstats":{"gc_next":160857440,"memory_alloc":81089160,"memory_total":937463480,"rss":-163840}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.08,"15":1.18,"5":1.18,"norm":{"1":1.08,"15":1.18,"5":1.18}}}}}}
2021-07-28T16:48:04.507+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:48:04.507+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1471 reconnect attempt(s)
2021-07-28T16:48:04.508+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:48:04.508+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:48:04.508+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:48:04.508+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:48:28.409+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8200,"time":{"ms":3}},"total":{"ticks":189760,"time":{"ms":7},"value":189760},"user":{"ticks":181560,"time":{"ms":4}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66360115}},"memstats":{"gc_next":160857440,"memory_alloc":81295896,"memory_total":937670216}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.99,"15":1.17,"5":1.15,"norm":{"1":0.99,"15":1.17,"5":1.15}}}}}}
2021-07-28T16:48:36.902+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:48:36.902+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1472 reconnect attempt(s)
2021-07-28T16:48:36.902+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:48:36.902+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:48:36.903+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:48:36.903+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:48:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8200,"time":{"ms":4}},"total":{"ticks":190060,"time":{"ms":302},"value":190060},"user":{"ticks":181860,"time":{"ms":298}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66390116}},"memstats":{"gc_next":160860992,"memory_alloc":80510888,"memory_total":938021760}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.04,"15":1.17,"5":1.14,"norm":{"1":1.04,"15":1.17,"5":1.14}}}}}}
2021-07-28T16:49:28.408+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8200,"time":{"ms":3}},"total":{"ticks":190060,"time":{"ms":5},"value":190060},"user":{"ticks":181860,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66420115}},"memstats":{"gc_next":160860992,"memory_alloc":80724136,"memory_total":938235008}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.94,"15":1.16,"5":1.11,"norm":{"1":0.94,"15":1.16,"5":1.11}}}}}}
2021-07-28T16:49:36.143+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:49:36.144+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1473 reconnect attempt(s)
2021-07-28T16:49:36.144+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:49:36.144+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:49:36.144+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:49:36.144+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:49:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8210,"time":{"ms":5}},"total":{"ticks":190080,"time":{"ms":8},"value":190080},"user":{"ticks":181870,"time":{"ms":3}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66450115}},"memstats":{"gc_next":160860992,"memory_alloc":81002376,"memory_total":938513248}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.24,"15":1.18,"5":1.16,"norm":{"1":1.24,"15":1.18,"5":1.16}}}}}}
2021-07-28T16:50:23.317+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:50:23.317+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1474 reconnect attempt(s)
2021-07-28T16:50:23.318+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:50:23.318+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:50:23.318+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:50:23.318+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:50:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8210,"time":{"ms":6}},"total":{"ticks":190080,"time":{"ms":6},"value":190080},"user":{"ticks":181870}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66480115}},"memstats":{"gc_next":160860992,"memory_alloc":81295336,"memory_total":938806208,"rss":-32768}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.37,"15":1.19,"5":1.19,"norm":{"1":1.37,"15":1.19,"5":1.19}}}}}}
2021-07-28T16:50:58.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8220,"time":{"ms":1}},"total":{"ticks":190360,"time":{"ms":276},"value":190360},"user":{"ticks":182140,"time":{"ms":275}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66510115}},"memstats":{"gc_next":160857440,"memory_alloc":80435936,"memory_total":939080592}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.14,"15":1.17,"5":1.15,"norm":{"1":1.14,"15":1.17,"5":1.15}}}}}}
2021-07-28T16:50:58.433+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:50:58.434+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1475 reconnect attempt(s)
2021-07-28T16:50:58.435+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:50:58.437+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:50:58.437+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:50:58.438+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:51:28.407+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8220,"time":{"ms":3}},"total":{"ticks":190370,"time":{"ms":6},"value":190370},"user":{"ticks":182150,"time":{"ms":3}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66540115}},"memstats":{"gc_next":160857440,"memory_alloc":80728600,"memory_total":939373256}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.34,"15":1.19,"5":1.2,"norm":{"1":1.34,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:51:43.820+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:51:43.820+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1476 reconnect attempt(s)
2021-07-28T16:51:43.820+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:51:43.821+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:51:43.821+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:51:43.821+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:58:46.307+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8220,"time":{"ms":5}},"total":{"ticks":190370,"time":{"ms":7},"value":190370},"user":{"ticks":182150,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66570116}},"memstats":{"gc_next":160857440,"memory_alloc":81167496,"memory_total":939812152}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.44,"15":1.2,"5":1.24,"norm":{"1":1.44,"15":1.2,"5":1.24}}}}}}
2021-07-28T16:59:16.308+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8230,"time":{"ms":4}},"total":{"ticks":190380,"time":{"ms":6},"value":190380},"user":{"ticks":182150,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66600115}},"memstats":{"gc_next":160857440,"memory_alloc":81287080,"memory_total":939931736}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.55,"15":1.22,"5":1.28,"norm":{"1":1.55,"15":1.22,"5":1.28}}}}}}
2021-07-28T16:59:30.978+0300    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:59:30.978+0300    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1477 reconnect attempt(s)
2021-07-28T16:59:30.978+0300    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2021-07-28T16:59:30.978+0300    INFO    [publish]   pipeline/retry.go:191     done
2021-07-28T16:59:30.978+0300    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2021-07-28T16:59:30.978+0300    INFO    [publish]   pipeline/retry.go:168     done
2021-07-28T16:59:46.307+0300    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8230,"time":{"ms":4}},"total":{"ticks":190670,"time":{"ms":297},"value":190670},"user":{"ticks":182440,"time":{"ms":293}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66630115}},"memstats":{"gc_next":160857408,"memory_alloc":80439184,"memory_total":940220744}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.3,"15":1.21,"5":1.25,"norm":{"1":1.3,"15":1.21,"5":1.25}}}}}}
ainayves commented 3 years ago

192.168.56.109:5044 is logstash endpoint

robcowart commented 3 years ago

You need to figure out why it can't connect to Logstash.

ainayves commented 3 years ago

You mean elasticsearch??? or filebeat???

robcowart commented 3 years ago

Filebeat. You can see the error in the Filebeat logs.

ainayves commented 3 years ago

So why : when I am doing a TCPDUMP on 5044 port in logstash ,, I am getting packets??

`sudo tcpdump -i enp0s8 -s 1500 port 5044 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s8, link-type EN10MB (Ethernet), capture size 1500 bytes

17:08:10.130365 IP 192.168.56.107.44528 > 192.168.56.109.5044: Flags [S], seq 2887510672, win 64240, options [mss 1460,sackOK,TS val 1683559313 ecr 0,nop,wscale 7], length 0

17:08:10.130389 IP 192.168.56.109.5044 > 192.168.56.107.44528: Flags [R.], seq 0, ack 2887510673, win 0, length 0 ` 192.168.56.107 is my suricata +filebeat server

robcowart commented 3 years ago

Just because packets are arriving doesn't mean that they a being received. Look at the logs... the details matter. It says "connection refused". It doesn't say "timed-out", or "no response"... it says the connection was REFUSED. Could be firewall, selinux, apparmor or any number of other things about your environment that are preventing a connection. That is what you have to figure out.

robcowart commented 3 years ago

Closing all issues as this project has been archived.