search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.49k stars 596 forks source link

docker image - flowcoll should recover once elasticsearch become available #710

Closed archont00 closed 3 years ago

archont00 commented 3 years ago

I use a separate container with elasticsearch & kibana. If a container based on elastiflow/flow-collector:5.0.1 is started before the elasticsearch cluster is up, it logs connection errors:

May 02 12:00:02 hostname flow-collector[835]: {"level":"error","ts":1619949602.9207108,"caller":"elasticsearch/pool.go:44","msg":"unable to put index template into elasticsearch cluster","code":"elasticsearch/invalid-index-template-request","reason":"dial tcp <ip_masked>:9201: connect: connection refused",...

The error continues even after the elasticsearch cluster has started and become available.

To resolve the bug, it is required to restart the flow-collector container.

The problem is quite significant when the server (or docker) is restarted - docker does not provide a reliable way to sort out the container dependencies.

Would it be possible to add a startup script waiting for elasticsearch cluster? Or anything similar.

https://docs.docker.com/compose/startup-order/

robcowart commented 3 years ago

I just tested this by bringing up the collector, seeing the errors and then bringing up elasticsearch. Once elasticsearch is available it connects fine.

{"level":"error","ts":1619953333.189947,"caller":"elasticsearch/pool.go:44","msg":"unable to put index template into elasticsearch cluster","code":"elasticsearch/invalid-index-template-request","reason":"EOF","stacktrace":"github.com/elastiflow/flowcoll/outputs/elasticsearch.(*Pool).handleElasticsearchIndexTemplate\n\t/Users/rob/go/src/github.com/elastiflow/flowcoll/outputs/elasticsearch/pool.go:44\ngithub.com/elastiflow/flowcoll/outputs/elasticsearch.(*Pool).Run\n\t/Users/rob/go/src/github.com/elastiflow/flowcoll/outputs/elasticsearch/pool.go:22"}
{"level":"error","ts":1619953344.210664,"caller":"elasticsearch/pool.go:44","msg":"unable to put index template into elasticsearch cluster","code":"elasticsearch/invalid-index-template-request","reason":"EOF","stacktrace":"github.com/elastiflow/flowcoll/outputs/elasticsearch.(*Pool).handleElasticsearchIndexTemplate\n\t/Users/rob/go/src/github.com/elastiflow/flowcoll/outputs/elasticsearch/pool.go:44\ngithub.com/elastiflow/flowcoll/outputs/elasticsearch.(*Pool).Run\n\t/Users/rob/go/src/github.com/elastiflow/flowcoll/outputs/elasticsearch/pool.go:22"}
{"level":"info","ts":1619953349.483244,"caller":"elasticsearch/elasticsearch.go:354","msg":"reading index template","source":"elasticsearch","version":"1.1-runtime"}
{"level":"info","ts":1619953349.483318,"caller":"elasticsearch/elasticsearch.go:52","msg":"running"}
robcowart commented 3 years ago

BTW, this is the repo for the legacy logstash-based version of ElastiFlow. Please join the community Slack for assistance with the new collector. The rest of the team hangs out there and you are likely to get help more quickly (you got lucky to catch me today).

archont00 commented 3 years ago

Okay, moving to Slack.