search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.72k stars 8.13k forks source link

[APM] Trace waterwall: add pagination to support big traces #164937

Closed sorenlouv closed 1 year ago

sorenlouv commented 1 year ago

Pitch: https://github.com/elastic/apm-dev/issues/1028

Problem

The waterfall is currently limited to displaying traces with up to 5000 items by default. It is possible to change this limit via xpack.apm.ui.maxTraceItems but there is a hard limit at 10.000 items. Going above this will break the waterfall entirely. The reason is that the Elasticsearch search API has a default limit for size at 10.000.

https://github.com/elastic/kibana/blob/9f83684f5fa5d0574d2547f55b78dc86a8c507d9/x-pack/plugins/apm/server/routes/traces/get_trace_items.ts#L109-L111

Solution

If maxTraceItems is higher than 10.000 we should paginate ES in batches of 10.000 to avoid breaking the ES limit. It's most likely much simpler to buffer the results on the server, instead of sending in chunks, so we should do that.

elasticmachine commented 1 year ago

Pinging @elastic/apm-ui (Team:APM)

flaper87 commented 1 year ago

We have recently been bitten by this. I literally wasted HOURS trying to find why multiple different services had exactly 10001 traces and why setting maxTraceItems to anything higher than 10k would break the waterfall. Now I know.

I don't want to get into the details of why we have so many traces, let's assume it's all good and expected. Would it be possible to please:

  1. Not hard-code the limit
  2. If you really must hard-code it, then document it (I looked for a while and I didn't find this documented anywhere, certainly not in the xpack plugin's docs)
  3. Maybe just allow for more than 10k transactions to be loaded? Use pagination if needed?
dgieselaar commented 1 year ago

@flaper87 you'll be happy to see the linked PR that @sqren has put up :) we will improve if further over time, good to an ex colleague using it. Appreciate the feedback!

sorenlouv commented 1 year ago

@flaper87 A fix will be available in 8.11 (and 8.10.1). I'd be very happy to hear you feedback if you happen to try this out. The default limit will still be 5000 but at least now you'll be able to increase it via xpack.apm.ui.maxTraceItems without hitting a hard limit.

flaper87 commented 1 year ago

Fantastic, thanks so much. Any idea when those will be out?

sorenlouv commented 1 year ago

I can't promise anything but 8.10.1 will likely be available next month.