Extend ELK monitoring for Logstash and Kibana

[joschi99]

It should be interessed to extend the monitoring of elasticsearch to monitor trough elastic api:

• Logstash nodes (CPU, memory, uptime, config reloads, events received, events emmited, events latency, ...) - filter for nodes
• Logstash pipelines (no. of pipelines, events emitted rate, no of nodes) - filter for nodes & pipelines
• Kibana (health status, Alerts, memory, max. response time, avg load, uptime) - filter for Kibana instance

Let me know if you need some sample datas

[garnier-quentin]

If you can provide curl commands and results, it will be great!

[joschi99]

Have done some research. Logstash provides a dedicated api for monitoring. There is a full description with curls and examples here: https://www.elastic.co/guide/en/logstash/7.13/node-stats-api.html

This should allow the check if logstash is working very well, uptime, memory and cpu usage and pipelines

For pipeline (https://www.elastic.co/guide/en/logstash/7.13/node-stats-api.html#pipeline-stats) is should be important to know if pipeline is runinng (I think otherwise should not present on output), and the events emitted rate, reload errors, queue type (memory or persited)

But I have to check how to reach the api from elasticsearch server,because the local logstash API will responde only to localhost

[joschi99]

It should be possible to connect with ssh and execute the curls locally? It should probably the most simply way

[joschi99]

for logstash is also important to monitor open files usage. If open_file_descriptors >= max_file_descriptors the process will stop

curl -XGET 'localhost:9600/_node/stats/process?pretty' { "host" : "loa-sr-ls01", "version" : "7.10.2", "http_address" : "127.0.0.1:9600", "id" : "8caa0e0c-a4e4-4617-b518-2220dfb432e1", "name" : "loa-sr-ls01", "ephemeral_id" : "38fb9e94-8819-407a-9924-c9f01a8523c8", "status" : "green", "snapshot" : false, "pipeline" : { "workers" : 4, "batch_size" : 125, "batch_delay" : 50 }, "monitoring" : { "hosts" : [ "https://loa-sr-es01:9200" ], "username" : "logstash_system" }, "process" : { "open_file_descriptors" : 754, "peak_open_file_descriptors" : 817, "max_file_descriptors" : 16384, "mem" : { "total_virtual_in_bytes" : 7640616960 }, "cpu" : { "total_in_millis" : 2052510, "percent" : 8, "load_average" : { "1m" : 0.68, "5m" : 1.75, "15m" : 3.06 } } } }

[Sims24]

@joschi99

Doing a SSH to execute locally a curl seems not very straightforward. It's not possible to access the API stats endpoints remotely ?

[joschi99]

Hi @Sims24, have done some further investigation. Yes, it should be possible to configure Logstash that we can reach the API from remote, so all requests described here (https://www.elastic.co/guide/en/logstash/7.13/monitoring-logstash.html) can be done without SSH.

[Sims24]

Thanks @joschi99

Last question, is there any authentication process/method to reach the API from a remote host? Or is it public?

Regards,

[joschi99]

There is no authentication, this is the reason why the endpoint normally is exposed only to 127.0.0.1 and not to a public interface.

[Sims24]

Ok that's a problem. We won't want to ask people to open potential security holes to monitor their Logstash.

We will do stg like: https://github.com/centreon/centreon-plugins/blob/2fd7bbb451f3dbf44b193d3566649bd0c2ed8c3e/apps/bluemind/local/mode/core.pm

It's officially in our internal backlog. Thanks

[joschi99]

There are only 2 options at the moment that Elastic offers to resolve this problem:

1. monitor logstash locally using 127.0.0.1 (through ssh?)
2. monitor logstash centrally through X-Pack configuration, where logstash send's his metrics and status to Elasticsearch

The second option has authentication from Elasticsearch, but requires configuration on Logstash, needs almost X-Pack basic, but could be possible. What do you think about them?

[Sims24]

The first one sounds like the less intrusive and with the better trade-off.

In this kind of situation, I think monitoring individually each component is more reliable than relying on additional configuration on third-party API.

[joschi99]

I agree with you. But to avoid a security problem could we execute the curl commands locally through SSH or maybe yo know a better approach?

[Sims24]

Yes, that is what we will do ;) The initial approach you proposed is the best one.

[joschi99]

Hi @Sims24, do you have some news about this integration?

[lucie-dubrunfaut]

Hello :)

This issue seems particularly old so let me close it. If the enhancement request is still current, please address it here to be take into account and vote on. Thank you for your understanding.