[APM] Surface all APM data anomalies in the curated APM UI

[nehaduggal]

Describe the feature:

With our out of the box integration we only surface the anomalies associated with the APM response time ML jobs in the curated APM experience. It'll be really helpful if we could add a way to surface any anomaly detected with APM data to surface in the curated APM experience in a generic way. For e.g. if a user chooses to add a ML job for error rate or for GC metrics, they should be able to see anomalies/health w.r.t to those jobs in the UI. As a part of this experience, we could also add a way for folks to pick what health indicators they choose to surface in the UI via a configuration and a way for them to navigate to the ML job to explore the detected anomalies.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

We might be able to achieve this if we can filter jobs by the index pattern they operate on. In our case we want to retrieve all jobs operating on apm-*. Is this possible @jgowdyelastic ?

[sorenlouv]

POC for getting the job id's for APM specific ML jobs:

GET .ml-config/_search
{
  "_source": "job_id", 
  "query": {
    "term": {
      "indices": "apm-*"
    }
  }
}

I realise we should use the ML client instead of querying directly. Would that be an okay approach?

[jgowdyelastic]

We do not currently have a kibana or es endpoint which can supply this functionality, but I think the best solution would be for ML to add a kibana endpoint to do this. It would need to first find datafeeds which match the index pattern and then find the related jobs. By us adding an endpoint/shared function to do this we'd only need to perform the capabilities and spaces checks once.

We shouldn't be searching the .ml-config index directly, the anomaly_detectors and datafeed endpoints should be the only way to retrieve job information.

EDIT: actually, if it's just the job id you're after, all you'd need to do is look through all of the datafeeds returned from <kibana>/api/ml/datafeeds as they contain job_id

[dgieselaar]

@sqren How do we decide what data to show where? We would need to be able to correlate service.name, transaction.type and possibly service.environment. Actually it would be nice if anomaly data was ECS compatible.

[sorenlouv]

How do we decide what data to show where? We would need to be able to correlate service.name, transaction.type and possibly service.environment. Actually it would be nice if anomaly data was ECS compatible.

Users would need to setup a multi-metric job and then specify the service.name in the "split field" option. It's error prone, and doesn't take transaction type or environment into account. It's very possible that our time is better spent extending the curated experience from within the APM ui. That being said, I wanted to entertain the thought and see if it was doable somehow.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.