x-pack monitoring metrics for ES rejects/retries and other perf metrics

[robgil]

While trying to identify the cause of poor ingest performance I was trying to find out the latency incurred by rejects and/or parsing exceptions on the ES side.

Ref: https://github.com/elastic/logstash/issues/5428 https://github.com/elastic/kibana/issues/18407

Here are a few metrics that would be helpful in troubleshooting performance issues like this.

• [ ] Number of bulk requests that have had rejects
• [ ] Number of events rejected over time
• [ ] Number of events hitting the DLQ
• [ ] bulk retries
• [ ] Latency of the bulk requests with rejects/retries taken into consideration
• [ ] Memory event buffer graphed over time
• [ ] Hot threads graphed by time spent
• [ ] Disk IO wait, especially with regards to the Persistent Queues (identify resource starvation on disk IO)
• [ ] Bucketed size of the bulk requests
  • I would like to identify small bulk requests and whether I need to adjust my delay to ensure a full bulk vs partial
• [ ] Full vs. partial bulk requests graph

[robgil]

One more

• [ ] The number of events being received by each thread => in the input {}
  • For some inputs we increase concurrency by increasing threads. I'd like to know whether this helps or hurts performance by seeing the ingest rate for each thread.