search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.7k stars 8.24k forks source link

[Stack Monitoring] Ingest pipeline monitoring #41936

Closed krisATelastic closed 1 year ago

krisATelastic commented 5 years ago

@joshdover has hijacked this issue and re-written most of the content below as of Oct 2022

User Story and requirements

Users need end to end monitoring of ingest pipelines in order to understand which ingest tier has performance bottlenecks and root cause expensive pipelines or processors. Specifically they're looking for the following features:

Nice to haves:

Existing metrics

Elasticsearch already exposes ingest pipeline metrics on the Nodes Stats API, including docs processed and CPU time on a pipeline and processor basis. These metrics are not currently ingested by our existing Metricbeat modules.

Proposed path forward

Stack Monitoring is in the middle of a transitional period which complicates how we make first steps here.

First, Stack Monitoring is not yet supported via Agent, only Metricbeat. This will be solved relatively soon by the Infra Monitoring UI team in https://github.com/elastic/kibana/issues/120415.

Second, there's been recent alignment and acceptance of an RFC to move towards an OTel-based solution for exposing metrics from stack components, away from Metricbeat modules. This is a great step forward that will allow teams to expose new metrics and have them visible in the UI with much less coordination across various teams and little to no UI development work required. This is still quite fresh and Elasticsearch does not yet support pushing OTel metrics to an OLTP endpoint, though it has recently added support for OTel traces.

[Done ✅] Phase 1: Add a new metricset to Metricbeat and add a dashboard to the upcoming Elasticsearch monitoring package for Elastic Agent

PRs:

We can likely get a basic solution out to help our customers quickly by building on top of the package being developed in https://github.com/elastic/kibana/issues/120415

This would involve:

The existing metricbeat module for es monitoring already polls these stats, but doesn't ingest any of the data. Enhancing Metricbeat directly with these stats has the benefit of being able to collect this data on Cloud deployments with the native "Logs and Metics" feature. This implementation would also be directly usable by the agent package.

Phase 2: Instrument Elasticsearch ingest pipelines w/ OTel traces and push to APM Server

Elasticsearch recently added support for shipping OTel traces to internal clusters for monitoring. This is not yet ready for end-user features, but would likely be a good fit for pipeline monitoring. A similar idea is discussed in https://github.com/elastic/kibana/issues/137141 for Logstash pipelines.

This phase would be in line with the long-term solution for Platform Observability, to use the OTel SDKs to push metrics to an OTLP endpoint, like APM Server.

One helpful piece of this is that instrumentating with traces that are pushed by Elasticsearch to an APM Server does not require Cloud to upgrade to Elastic Agent at the same time.

This phase would involve:

Phase 3: Add OTel metrics to Elasticsearch

In this phase we would add high-level ingest pipeline metrics that are exported via OTel, rather than the Node Stats API. These would replace the stop-gap solution introduced in phase 1 by allowing us to add more, richer metrics without requiring changes to the Stack Monitoring UI and Metricbeat modules.

This phase would involve:

Worklist - outdated

References

Original issue description **Describe the feature:** Elasticsearch Ingest node monitoring **Describe a specific use case for the feature:** As more native tools like Beats are being pushed to Ingest nodes as ingest pipelines are tied to some of the developed modules, it would be good to expand Monitoring within Kibana to include the ingest pipelines statistics from node stats, i.e. `GET _nodes/stats?filter_path=nodes.*.ingest.*`. This responds with ``` { "nodes" : { "JokQCpQpTAmZLQD9XjwbAA" : { "ingest" : { "total" : { "count" : 0, "time_in_millis" : 0, "current" : 0, "failed" : 0 }, "pipelines" : { "xpack_monitoring_6" : { "count" : 0, "time_in_millis" : 0, "current" : 0, "failed" : 0, "processors" : [ { "script" : { "count" : 0, "time_in_millis" : 0, "current" : 0, "failed" : 0 } }, { "gsub" : { "count" : 0, "time_in_millis" : 0, "current" : 0, "failed" : 0 } } ] }, "xpack_monitoring_7" : { "count" : 0, "time_in_millis" : 0, "current" : 0, "failed" : 0, "processors" : [ ] } } } } } } ``` Logstash has pipeline monitoring, it would be good to also have the same feature for Ingest.
elasticmachine commented 5 years ago

Pinging @elastic/es-ui

elasticmachine commented 5 years ago

Pinging @elastic/stack-monitoring

cachedout commented 5 years ago

Hi @krisATelastic. Thanks for this suggestion! We've had this requested a number of times and we're definitely looking into the feasibility.

krisATelastic commented 5 years ago

That's great it's already on the radar! For housekeeping, quite happy to +1 the original issue and close this one out, if that is easiest for the team.

djmcgreal-cc commented 4 years ago

Hi. Firstly thanks for all the good work so far! It seems like a gap in 360 monitoring to not be able to get something similar from ingest nodes. Is there a workaround I'm missing? Thanks, Dan.

tadgh commented 3 years ago

Just piling on here to say I'd love to see this added. The stats API is fine for reading it, but would be nice to see it in kibana monitoring

cjcenizal commented 3 years ago

Related ES request: https://github.com/elastic/elasticsearch/issues/43368

elasticmachine commented 2 years ago

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

joshdover commented 2 years ago

I've updated this issue based on the most recent information I have gathered in the past week after discussing with several folks.

cc @mukeshelastic @amitkanfer @ruflin @skearns64 @tylerperk @qhoxie who were involved in recent discussions.

joshdover commented 2 years ago

Another appealing option as highlighted to me by @miltonhultgren would be to start by tracing ingest pipelines using Elasticsearch's new tracing support which traces ships to APM Server and are viewable in the APM UI. The same solution was suggested for Logstash pipelines in https://github.com/elastic/kibana/issues/137141.

amitkanfer commented 2 years ago

@joshdover do we want to have metrics around ingest volume? (in bytes)

andrewkroh commented 2 years ago

This phase would involve:

Adding trace instrumentation to Elasticsearch ingest pipeline code, incl metrics like CPU time and bytes/docs processed.

This might be out of scope for this issue. With respect to tracing, one of the questions we would likely by trying to answer is "why is my _bulk request taking so long?" If the tracing could start at the bulk request it would help us to answer that. Potentially this would give us visibility into decompression time, how long the request waited in the write threadpool queue, json deserialization time, etc.

joshdover commented 1 year ago

A beta for phase 1 is shipping in 8.7, enabled by default for Cloud customers. Further explorations of other phases or implementations should be tracked in separate issues.