[APM] Internal Server Error returns on apm/traces/aggregated_critical request

[ablnk]

Version: Serverless project v 8.14.0 Stateful deployment v 8.14.0-SNAPSHOT

Description: POST /internal/apm/traces/aggregated_critical_path request return Internal Server Error.

Preconditions: I reproduced the issue having ~780k documents in APM data view within 15 minutes interval from 761 services.

Steps to reproduce:

1. Go to Applications - Traces - Explorer.
2. Ensure the date picker is set to at least 15 minutes interval.
3. Click Aggregated critical path tab.

Expected behavior: Data presentation should be rendered.

Screenshots:

https://github.com/elastic/kibana/assets/34958359/0f201a48-e600-492f-8ce5-31ec25f19859

Response:

{
    "statusCode": 500,
    "error": "Internal Server Error",
    "message": "{\"ok\":false,\"message\":\"The instance rejected the connection.\"}",
    "attributes": {
        "data": {},
        "_inspect": [
            {
                "id": "get_aggregated_critical_path (/internal/apm/traces/aggregated_critical_path) a03c86a1-1fce-444d-8b11-44078efe4ce1",
                "json": {
                    "size": 0,
                    "track_total_hits": false,
                    "query": {
                        "bool": {
                            "filter": [
                                {
                                    "terms": {
                                        "processor.event": [
                                            "span",
                                            "transaction"
                                        ]
                                    }
                                }
                            ],
                            "must": [
                                {
                                    "bool": {
                                        "filter": [
                                            {
                                                "terms": {
                                                    "trace.id": [
                                                        "00000000000000000000000000010895",
                                                        "00000000000000000000000000010973",
                                                        "00000000000000000000000000006371",
                                                        "00000000000000000000000000007463",
                                                        "00000000000000000000000000011051",
                                                        "00000000000000000000000000011987",
                                                        "00000000000000000000000000012143",
                                                        "00000000000000000000000000013313",
                                                        "00000000000000000000000000005279",
                                                        "00000000000000000000000000005825",
                                                        "00000000000000000000000000007073",
                                                        "00000000000000000000000000007619",
                                                        "00000000000000000000000000008009",
                                                        "00000000000000000000000000010427",
                                                        "00000000000000000000000000010583",
                                                        "00000000000000000000000000011129",
                                                        "00000000000000000000000000011285",
                                                        "00000000000000000000000000011363",
                                                        "00000000000000000000000000011441",
                                                        "00000000000000000000000000011597",
                                                        "00000000000000000000000000012065",
                                                        "00000000000000000000000000013235",
                                                        "00000000000000000000000000014249",
                                                        "00000000000000000000000000014405",
                                                        "00000000000000000000000000004265",
                                                        "00000000000000000000000000004733",
                                                        "00000000000000000000000000004811",
                                                        "00000000000000000000000000005513",
                                                        "00000000000000000000000000006917",
                                                        "00000000000000000000000000008789",
                                                        "00000000000000000000000000009101",
                                                        "00000000000000000000000000009179",
                                                        "00000000000000000000000000009257",
                                                        "00000000000000000000000000009335",
                                                        "00000000000000000000000000009413",
                                                        "00000000000000000000000000009725",
                                                        "00000000000000000000000000009803",
                                                        "00000000000000000000000000009881",
                                                        "00000000000000000000000000010115",
                                                        "00000000000000000000000000010193",
                                                        "00000000000000000000000000010349",
                                                        "00000000000000000000000000010505",
                                                        "00000000000000000000000000010661",
                                                        "00000000000000000000000000010739",
                                                        "00000000000000000000000000010817",
                                                        "00000000000000000000000000011207",
                                                        "00000000000000000000000000011753",
                                                        "00000000000000000000000000012221",
                                                        "00000000000000000000000000012377",
                                                        "00000000000000000000000000012455",
                                                        "00000000000000000000000000012767",
                                                        "00000000000000000000000000013079",
                                                        "00000000000000000000000000013157",
                                                        "00000000000000000000000000013391",
                                                        "00000000000000000000000000013859",
                                                        "00000000000000000000000000014327",
                                                        "00000000000000000000000000003953",
                                                        "00000000000000000000000000004187",
                                                        "00000000000000000000000000004343",
                                                        "00000000000000000000000000005201",
                                                        "00000000000000000000000000005747",
                                                        "00000000000000000000000000005981",
                                                        "00000000000000000000000000006059",
                                                        "00000000000000000000000000006449",
                                                        "00000000000000000000000000006527",
                                                        "00000000000000000000000000007541",
                                                        "00000000000000000000000000008165",
                                                        "00000000000000000000000000008633",
                                                        "00000000000000000000000000008711",
                                                        "00000000000000000000000000009491",
                                                        "00000000000000000000000000009569",
                                                        "00000000000000000000000000009647",
                                                        "00000000000000000000000000010037",
                                                        "00000000000000000000000000010271",
                                                        "00000000000000000000000000011519",
                                                        "00000000000000000000000000011675",
                                                        "00000000000000000000000000011831",
                                                        "00000000000000000000000000011909",
                                                        "00000000000000000000000000012299",
                                                        "00000000000000000000000000012533",
                                                        "00000000000000000000000000012611",
                                                        "00000000000000000000000000012689",
                                                        "00000000000000000000000000012845",
                                                        "00000000000000000000000000012923",
                                                        "00000000000000000000000000013001",
                                                        "00000000000000000000000000013703",
                                                        "00000000000000000000000000014015",
                                                        "00000000000000000000000000014093",
                                                        "00000000000000000000000000014483",
                                                        "00000000000000000000000000014561",
                                                        "00000000000000000000000000015575",
                                                        "00000000000000000000000000015757",
                                                        "00000000000000000000000000016121",
                                                        "00000000000000000000000000003407",
                                                        "00000000000000000000000000004109",
                                                        "00000000000000000000000000004655",
                                                        "00000000000000000000000000004967",
                                                        "00000000000000000000000000005123",
                                                        "00000000000000000000000000005357",
                                                        "00000000000000000000000000009959",
                                                        "00000000000000000000000000013105",
                                                        "00000000000000000000000000013469",
                                                        "00000000000000000000000000013781",
                                                        "00000000000000000000000000013937",
                                                        "00000000000000000000000000014171",
                                                        "00000000000000000000000000014665",
                                                        "00000000000000000000000000014821",
                                                        "00000000000000000000000000015419",
                                                        "00000000000000000000000000015497",
                                                        "00000000000000000000000000015913",
                                                        "00000000000000000000000000005461",
                                                        "00000000000000000000000000007645",
                                                        "00000000000000000000000000013547",
                                                        "00000000000000000000000000013573",
                                                        "00000000000000000000000000013625",
                                                        "00000000000000000000000000014639",
                                                        "00000000000000000000000000014717",
                                                        "00000000000000000000000000014795",
                                                        "00000000000000000000000000014873",
                                                        "00000000000000000000000000015029",
                                                        "00000000000000000000000000015107",
                                                        "00000000000000000000000000015263",
                                                        "00000000000000000000000000015653",
                                                        "00000000000000000000000000015809",
                                                        "00000000000000000000000000015887",
                                                        "00000000000000000000000000015965",
                                                        "00000000000000000000000000016043",
                                                        "00000000000000000000000000016225",
                                                        "00000000000000000000000000020281",
                                                        "00000000000000000000000000003433",
                                                        "00000000000000000000000000006553",
                                                        "00000000000000000000000000006709",
                                                        "00000000000000000000000000006865",
                                                        "00000000000000000000000000007281",
                                                        "00000000000000000000000000007801",
                                                        "00000000000000000000000000007957",
                                                        "00000000000000000000000000013885",
                                                        "00000000000000000000000000014197",
                                                        "00000000000000000000000000014509",
                                                        "00000000000000000000000000014951",
                                                        "00000000000000000000000000015185",
                                                        "00000000000000000000000000015289",
                                                        "00000000000000000000000000015341",
                                                        "00000000000000000000000000015445",
                                                        "00000000000000000000000000015601",
                                                        "00000000000000000000000000015731",
                                                        "00000000000000000000000000015835",
                                                        "00000000000000000000000000016069",
                                                        "00000000000000000000000000016199",
                                                        "00000000000000000000000000016381",
                                                        "00000000000000000000000000016537",
                                                        "00000000000000000000000000018097",
                                                        "00000000000000000000000000019189",
                                                        "00000000000000000000000000019345",
                                                        "00000000000000000000000000020437",
                                                        "00000000000000000000000000021529",
                                                        "00000000000000000000000000027223",
                                                        "00000000000000000000000000004369",
                                                        "00000000000000000000000000004525",
                                                        "00000000000000000000000000005773",
                                                        "00000000000000000000000000009049",
                                                        "00000000000000000000000000011389",
                                                        "00000000000000000000000000012481",
                                                        "00000000000000000000000000012637",
                                                        "00000000000000000000000000012793",
                                                        "00000000000000000000000000012949",
                                                        "00000000000000000000000000013261",
                                                        "00000000000000000000000000013417",
                                                        "00000000000000000000000000013729",
                                                        "00000000000000000000000000014041",
                                                        "00000000000000000000000000014353",
                                                        "00000000000000000000000000014977",
                                                        "00000000000000000000000000015133",
                                                        "00000000000000000000000000015367",
                                                        "00000000000000000000000000016511",
                                                        "00000000000000000000000000018253",
                                                        "00000000000000000000000000018721",
                                                        "00000000000000000000000000020749",
                                                        "00000000000000000000000000021061",
                                                        "00000000000000000000000000024415",
                                                        "00000000000000000000000000024961",
                                                        "00000000000000000000000000025975",
                                                        "00000000000000000000000000026053",
                                                        "00000000000000000000000000026131",
                                                        "00000000000000000000000000026677",
                                                        "00000000000000000000000000027067",
                                                        "00000000000000000000000000027145",
                                                        "00000000000000000000000000010297",
                                                        "00000000000000000000000000011233",
                                                        "00000000000000000000000000013495",
                                                        "00000000000000000000000000013651",
                                                        "00000000000000000000000000014743",
                                                        "00000000000000000000000000015523",
                                                        "00000000000000000000000000015679",
                                                        "00000000000000000000000000016147",
                                                        "00000000000000000000000000016433",
                                                        "00000000000000000000000000018409",
                                                        "00000000000000000000000000018565",
                                                        "00000000000000000000000000018877",
                                                        "00000000000000000000000000019033",
                                                        "00000000000000000000000000019501",
                                                        "00000000000000000000000000019657",
                                                        "00000000000000000000000000019813",
                                                        "00000000000000000000000000019969",
                                                        "00000000000000000000000000020125",
                                                        "00000000000000000000000000020359",
                                                        "00000000000000000000000000020593",
                                                        "00000000000000000000000000020905",
                                                        "00000000000000000000000000021217",
                                                        "00000000000000000000000000021373",
                                                        "00000000000000000000000000021607",
                                                        "00000000000000000000000000022699",
                                                        "00000000000000000000000000023713",
                                                        "00000000000000000000000000023791",
                                                        "00000000000000000000000000023869",
                                                        "00000000000000000000000000024805",
                                                        "00000000000000000000000000024883",
                                                        "00000000000000000000000000026365",
                                                        "00000000000000000000000000027379",
                                                        "00000000000000000000000000027613",
                                                        "00000000000000000000000000028315",
                                                        "00000000000000000000000000031253",
                                                        "00000000000000000000000000010557",
                                                        "00000000000000000000000000010609",
                                                        "00000000000000000000000000010765",
                                                        "00000000000000000000000000010921",
                                                        "00000000000000000000000000011311",
                                                        "00000000000000000000000000011545",
                                                        "00000000000000000000000000011857",
                                                        "00000000000000000000000000012169",
                                                        "00000000000000000000000000012325",
                                                        "00000000000000000000000000012403",
                                                        "00000000000000000000000000012559",
                                                        "00000000000000000000000000012715",
                                                        "00000000000000000000000000012819",
                                                        "00000000000000000000000000012897",
                                                        "00000000000000000000000000013027",
                                                        "00000000000000000000000000013183",
                                                        "00000000000000000000000000013339",
                                                        "00000000000000000000000000013911",
                                                        "00000000000000000000000000013989",
                                                        "00000000000000000000000000014067",
                                                        "00000000000000000000000000014119",
                                                        "00000000000000000000000000014431",
                                                        "00000000000000000000000000014613",
                                                        "00000000000000000000000000015003",
                                                        "00000000000000000000000000015055",
                                                        "00000000000000000000000000015159",
                                                        "00000000000000000000000000015991",
                                                        "00000000000000000000000000016173",
                                                        "00000000000000000000000000016251",
                                                        "00000000000000000000000000016277",
                                                        "00000000000000000000000000016303",
                                                        "00000000000000000000000000016329",
                                                        "00000000000000000000000000016459",
                                                        "00000000000000000000000000017785",
                                                        "00000000000000000000000000017941",
                                                        "00000000000000000000000000025507",
                                                        "00000000000000000000000000025585",
                                                        "00000000000000000000000000025663",
                                                        "00000000000000000000000000025741",
                                                        "00000000000000000000000000025819",
                                                        "00000000000000000000000000025897",
                                                        "00000000000000000000000000026209",
                                                        "00000000000000000000000000026287",
                                                        "00000000000000000000000000026443",
                                                        "00000000000000000000000000026521",
                                                        "00000000000000000000000000026599",
                                                        "00000000000000000000000000026755",
                                                        "00000000000000000000000000026833",
                                                        "00000000000000000000000000026885",
                                                        "00000000000000000000000000026911",
                                                        "00000000000000000000000000026989",
                                                        "00000000000000000000000000027301",
                                                        "00000000000000000000000000027457",
                                                        "00000000000000000000000000027535",
                                                        "00000000000000000000000000027691",
                                                        "00000000000000000000000000027769",
                                                        "00000000000000000000000000027847",
                                                        "00000000000000000000000000027925",
                                                        "00000000000000000000000000028237",
                                                        "00000000000000000000000000028653",
                                                        "00000000000000000000000000029485",
                                                        "00000000000000000000000000030161",
                                                        "00000000000000000000000000030239",
                                                        "00000000000000000000000000031201",
                                                        "00000000000000000000000000031279",
                                                        "00000000000000000000000000031409",
                                                        "00000000000000000000000000031591",
                                                        "00000000000000000000000000031669",
                                                        "00000000000000000000000000003993",
                                                        "00000000000000000000000000007989",
                                                        "00000000000000000000000000010453",
                                                        "00000000000000000000000000010479",
                                                        "00000000000000000000000000010635",
                                                        "00000000000000000000000000010843",
                                                        "00000000000000000000000000011077",
                                                        "00000000000000000000000000011649",
                                                        "00000000000000000000000000011701",
                                                        "00000000000000000000000000011727",
                                                        "00000000000000000000000000011805",
                                                        "00000000000000000000000000012013",
                                                        "00000000000000000000000000012351",
                                                        "00000000000000000000000000012741",
                                                        "00000000000000000000000000012871",
                                                        "00000000000000000000000000013807",
                                                        "00000000000000000000000000013963",
                                                        "00000000000000000000000000014275",
                                                        "00000000000000000000000000014587",
                                                        "00000000000000000000000000014899",
                                                        "00000000000000000000000000015081",
                                                        "00000000000000000000000000015211",
                                                        "00000000000000000000000000016355",
                                                        "00000000000000000000000000025559",
                                                        "00000000000000000000000000025637",
                                                        "00000000000000000000000000025715",
                                                        "00000000000000000000000000025923",
                                                        "00000000000000000000000000026313",
                                                        "00000000000000000000000000026391",
                                                        "00000000000000000000000000026469",
                                                        "00000000000000000000000000026729",
                                                        "00000000000000000000000000026807",
                                                        "00000000000000000000000000027405",
                                                        "00000000000000000000000000027431",
                                                        "00000000000000000000000000027483",
                                                        "00000000000000000000000000027561",
                                                        "00000000000000000000000000027639",
                                                        "00000000000000000000000000027821",
                                                        "00000000000000000000000000027873",
                                                        "00000000000000000000000000027899",
                                                        "00000000000000000000000000027977",
                                                        "00000000000000000000000000028393",
                                                        "00000000000000000000000000028575",
                                                        "00000000000000000000000000028731",
                                                        "00000000000000000000000000028939",
                                                        "00000000000000000000000000028991",
                                                        "00000000000000000000000000029069",
                                                        "00000000000000000000000000029147",
                                                        "00000000000000000000000000029173",
                                                        "00000000000000000000000000029225",
                                                        "00000000000000000000000000029329",
                                                        "00000000000000000000000000029381",
                                                        "00000000000000000000000000029407",
                                                        "00000000000000000000000000029693",
                                                        "00000000000000000000000000029719",
                                                        "00000000000000000000000000029875",
                                                        "00000000000000000000000000030005",
                                                        "00000000000000000000000000030031",
                                                        "00000000000000000000000000030083",
                                                        "00000000000000000000000000030317",
                                                        "00000000000000000000000000030343",
                                                        "00000000000000000000000000030499",
                                                        "00000000000000000000000000030577",
                                                        "00000000000000000000000000030655",
                                                        "00000000000000000000000000030733",
                                                        "00000000000000000000000000030811",
                                                        "00000000000000000000000000030889",
                                                        "00000000000000000000000000030915",
                                                        "00000000000000000000000000030941",
                                                        "00000000000000000000000000030967",
                                                        "00000000000000000000000000030993",
                                                        "00000000000000000000000000031045",
                                                        "00000000000000000000000000031097",
                                                        "00000000000000000000000000031123",
                                                        "00000000000000000000000000031175",
                                                        "00000000000000000000000000031331",
                                                        "00000000000000000000000000031357",
                                                        "00000000000000000000000000031435",
                                                        "00000000000000000000000000031513",
                                                        "00000000000000000000000000005991",
                                                        "00000000000000000000000000009987",
                                                        "00000000000000000000000000015981",
                                                        "00000000000000000000000000023973",
                                                        "00000000000000000000000000025767",
                                                        "00000000000000000000000000025793",
                                                        "00000000000000000000000000026027",
                                                        "00000000000000000000000000026339",
                                                        "00000000000000000000000000026573",
                                                        "00000000000000000000000000026625",
                                                        "00000000000000000000000000026781",
                                                        "00000000000000000000000000027093",
                                                        "00000000000000000000000000027249",
                                                        "00000000000000000000000000027275",
                                                        "00000000000000000000000000027353",
                                                        "00000000000000000000000000027743",
                                                        "00000000000000000000000000028029",
                                                        "00000000000000000000000000028107",
                                                        "00000000000000000000000000028185",
                                                        "00000000000000000000000000028211",
                                                        "00000000000000000000000000028341",
                                                        "00000000000000000000000000029043",
                                                        "00000000000000000000000000029277",
                                                        "00000000000000000000000000029745",
                                                        "00000000000000000000000000029823",
                                                        "00000000000000000000000000029901",
                                                        "00000000000000000000000000030135",
                                                        "00000000000000000000000000030447",
                                                        "00000000000000000000000000030837",
                                                        "00000000000000000000000000031071",
                                                        "00000000000000000000000000031149",
                                                        "00000000000000000000000000073923",
                                                        "00000000000000000000000000001995",
                                                        "00000000000000000000000000035961",
                                                        "00000000000000000000000000041955",
                                                        "00000000000000000000000000046721",
                                                        "00000000000000000000000000061935",
                                                        "00000000000000000000000000067929",
                                                        "00000000000000000000000000087909",
                                                        "00000000000000000000000000093903",
                                                        "00000000000000000000000000099897",
                                                        "00000000000000000000000000157839",
                                                        "00000000000000000000000000167829",
                                                        "00000000000000000000000000191805",
                                                        "00000000000000000000000000011985",
                                                        "00000000000000000000000000013983",
                                                        "00000000000000000000000000027969",
                                                        "00000000000000000000000000029967",
                                                        "00000000000000000000000000043953",
                                                        "00000000000000000000000000045951",
                                                        "00000000000000000000000000047949",
                                                        "00000000000000000000000000053943",
                                                        "00000000000000000000000000057939",
                                                        "00000000000000000000000000077919",
                                                        "00000000000000000000000000081915",
                                                        "00000000000000000000000000089907",
                                                        "00000000000000000000000000091905",
                                                        "00000000000000000000000000101895",
                                                        "00000000000000000000000000133863",
                                                        "00000000000000000000000000135861",
                                                        "00000000000000000000000000141855",
                                                        "00000000000000000000000000143853",
                                                        "00000000000000000000000000145851",
                                                        "00000000000000000000000000147849",
                                                        "00000000000000000000000000149847",
                                                        "00000000000000000000000000151845",
                                                        "00000000000000000000000000153843",
                                                        "00000000000000000000000000155841",
                                                        "00000000000000000000000000159837",
                                                        "00000000000000000000000000161835",
                                                        "00000000000000000000000000163833",
                                                        "00000000000000000000000000165831",
                                                        "00000000000000000000000000169827",
                                                        "00000000000000000000000000171825",
                                                        "00000000000000000000000000173823",
                                                        "00000000000000000000000000175821",
                                                        "00000000000000000000000000177819",
                                                        "00000000000000000000000000179817",
                                                        "00000000000000000000000000181815",
                                                        "00000000000000000000000000183813",
                                                        "00000000000000000000000000185811",
                                                        "00000000000000000000000000187809",
                                                        "00000000000000000000000000189807",
                                                        "00000000000000000000000000193803",
                                                        "00000000000000000000000000195801",
                                                        "00000000000000000000000000197799",
                                                        "00000000000000000000000000199797",
                                                        "00000000000000000000000000373775",
                                                        "00000000000000000000000000420497",
                                                        "00000000000000000000000000017979",
                                                        "00000000000000000000000000019977",
                                                        "00000000000000000000000000021975",
                                                        "00000000000000000000000000025971",
                                                        "00000000000000000000000000031965",
                                                        "00000000000000000000000000033963",
                                                        "00000000000000000000000000037959",
                                                        "00000000000000000000000000039957",
                                                        "00000000000000000000000000049947",
                                                        "00000000000000000000000000051945",
                                                        "00000000000000000000000000055941",
                                                        "00000000000000000000000000059937",
                                                        "00000000000000000000000000063933",
                                                        "00000000000000000000000000065931",
                                                        "00000000000000000000000000069927",
                                                        "00000000000000000000000000071925",
                                                        "00000000000000000000000000075921",
                                                        "00000000000000000000000000079917",
                                                        "00000000000000000000000000083913",
                                                        "00000000000000000000000000085911",
                                                        "00000000000000000000000000095901",
                                                        "00000000000000000000000000097899",
                                                        "00000000000000000000000000103893",
                                                        "00000000000000000000000000105891",
                                                        "00000000000000000000000000107889",
                                                        "00000000000000000000000000109887",
                                                        "00000000000000000000000000111885",
                                                        "00000000000000000000000000113883",
                                                        "00000000000000000000000000115881",
                                                        "00000000000000000000000000117879",
                                                        "00000000000000000000000000119877",
                                                        "00000000000000000000000000121875",
                                                        "00000000000000000000000000123873",
                                                        "00000000000000000000000000125871",
                                                        "00000000000000000000000000127869",
                                                        "00000000000000000000000000129867",
                                                        "00000000000000000000000000131865",
                                                        "00000000000000000000000000137859",
                                                        "00000000000000000000000000139857",
                                                        "00000000000000000000000000280331",
                                                        "00000000000000000000000000327053",
                                                        "00000000000000000000000000467219"
                                                    ]
                                                }
                                            },
                                            {
                                                "range": {
                                                    "@timestamp": {
                                                        "gte": 1710606359591,
                                                        "lte": 1710952859591,
                                                        "format": "epoch_millis"
                                                    }
                                                }
                                            }
                                        ]
                                    }
                                }
                            ]
                        }
                    },
                    "aggs": {
                        "critical_path": {
                            "scripted_metric": {
                                "params": {},
                                "init_script": {
                                    "source": "\n                state.eventsById = [:];\n                state.metadataByOperationId = [:];\n              "
                                },
                                "map_script": {
                                    "source": "\n                String toHash (def item) {\n                  long FNV_32_INIT = 0x811c9dc5L;\n                  long FNV_32_PRIME = 0x01000193L;\n                  char[] chars = item.toString().toCharArray();\n                  long rv = FNV_32_INIT;\n                  int len = chars.length;\n                  for(int i = 0; i < len; i++) {\n                      byte bt = (byte) chars[i];\n                      rv ^= bt;\n                      rv *= FNV_32_PRIME;\n                  }\n                  return rv.toString();\n                }\n                \n                def id;\n                double duration;\n                \n                def operationMetadata = [\n                  \"service.name\": doc['service.name'].value,\n                  \"processor.event\": doc['processor.event'].value,\n                  \"agent.name\": doc['agent.name'].value\n                ];\n\n                def isSpan = !doc['span.id'].empty && !doc['span.name'].empty;\n                \n                if (isSpan) {\n                  id = doc['span.id'].value;\n                  operationMetadata.put('span.name', doc['span.name'].value);\n                  if (!doc['span.type'].empty) {\n                    operationMetadata.put('span.type', doc['span.type'].value);\n                  }\n                  if (!doc['span.subtype'].empty) {\n                    operationMetadata.put('span.subtype', doc['span.subtype'].value);\n                  }\n                  duration = doc['span.duration.us'].value;\n                } else {\n                  id = doc['transaction.id'].value;\n                  operationMetadata.put('transaction.name', doc['transaction.name'].value);\n                  operationMetadata.put('transaction.type', doc['transaction.type'].value);\n                  duration = doc['transaction.duration.us'].value;\n                }\n                 \n                String operationId = toHash(operationMetadata);\n                \n                def map = [\n                  \"traceId\": doc['trace.id'].value,\n                  \"id\": id,\n                  \"parentId\": doc['parent.id'].empty ? null : doc['parent.id'].value,\n                  \"operationId\": operationId,\n                  \"timestamp\": doc['timestamp.us'].value,\n                  \"duration\": duration\n                ];\n                \n                if (state.metadataByOperationId[operationId] == null) {\n                  state.metadataByOperationId.put(operationId, operationMetadata);\n                }\n                state.eventsById.put(id, map);\n              "
                                },
                                "combine_script": {
                                    "source": "return state;"
                                },
                                "reduce_script": {
                                    "source": "\n                String toHash (def item) {\n                  long FNV_32_INIT = 0x811c9dc5L;\n                  long FNV_32_PRIME = 0x01000193L;\n                  char[] chars = item.toString().toCharArray();\n                  long rv = FNV_32_INIT;\n                  int len = chars.length;\n                  for(int i = 0; i < len; i++) {\n                      byte bt = (byte) chars[i];\n                      rv ^= bt;\n                      rv *= FNV_32_PRIME;\n                  }\n                  return rv.toString();\n                }\n                \n                def processEvent (def context, def event) {\n                  if (context.processedEvents[event.id] != null) {\n                    return context.processedEvents[event.id];\n                  }\n                  \n                  def processedEvent = [\n                    \"children\": []\n                  ];\n                  \n                  if(event.parentId != null) {\n                    def parent = context.events[event.parentId];\n                    if (parent == null) {\n                      return null;\n                    }\n                    def processedParent = processEvent(context, parent);\n                    if (processedParent == null) {\n                      return null;\n                    }\n                    processedParent.children.add(processedEvent);\n                  }\n                  \n                  context.processedEvents.put(event.id, processedEvent);\n                  \n                  processedEvent.putAll(event);\n\n                  if (context.params.serviceName != null && context.params.transactionName != null) {\n                    \n                    def metadata = context.metadata[event.operationId];\n                    \n                    if (metadata != null\n                      && context.params.serviceName == metadata['service.name']\n                      && metadata['transaction.name'] != null \n                      && context.params.transactionName == metadata['transaction.name']\n                    ) {\n                      context.entryTransactions.add(processedEvent);\n                    }\n\n                  } else if (event.parentId == null) {\n                    context.entryTransactions.add(processedEvent);\n                  }\n                  \n                  return processedEvent;\n                }\n                \n                double getClockSkew (def context, def item, def parent ) {\n                  if (parent == null) {\n                    return 0;\n                  }\n                  \n                  def processorEvent = context.metadata[item.operationId]['processor.event'];\n                  \n                  def isTransaction = processorEvent == 'transaction';\n                  \n                  if (!isTransaction) {\n                    return parent.skew;\n                  }\n                  \n                  double parentStart = parent.timestamp + parent.skew;\n                  double offsetStart = parentStart - item.timestamp;\n                  if (offsetStart > 0) {\n                    double latency = Math.round(Math.max(parent.duration - item.duration, 0) / 2);\n                    return offsetStart + latency;\n                  }\n                  \n                  return 0;\n                }\n                \n                void setOffsetAndSkew ( def context, def event, def parent, def startOfTrace ) {\n                  event.skew = getClockSkew(context, event, parent);\n                  event.offset = event.timestamp - startOfTrace;\n                  for(child in event.children) {\n                    setOffsetAndSkew(context, child, event, startOfTrace);\n                  }\n                  event.end = event.offset + event.skew + event.duration;\n                }\n                \n                void count ( def context, def nodeId, def duration ) {\n                  context.timeByNodeId[nodeId] = (context.timeByNodeId[nodeId] ?: 0) + duration;\n                }\n                \n                void scan ( def context, def item, def start, def end, def path ) {\n                  \n                  def nodeId = toHash(path);\n        \n                  def childNodes = context.nodes[nodeId] != null ? context.nodes[nodeId] : [];\n                  \n                  context.nodes[nodeId] = childNodes;\n                  \n                  context.operationIdByNodeId[nodeId] = item.operationId;\n                  \n                  if (item.children.size() == 0) {\n                    count(context, nodeId, end - start);\n                    return;\n                  }\n                  \n                  item.children.sort((a, b) -> {\n                    if (b.end === a.end) {\n                      return 0;\n                    }\n                    if (b.end > a.end) {\n                      return 1;\n                    }\n                    return -1;\n                  });\n                  \n                  def scanTime = end;\n                  \n                  for(child in item.children) {\n                    double normalizedChildStart = Math.max(child.offset + child.skew, start);\n                    double childEnd = child.offset + child.skew + child.duration;\n                    \n                    double normalizedChildEnd = Math.min(childEnd, scanTime);\n              \n                    def isOnCriticalPath = !(\n                      normalizedChildStart >= scanTime ||\n                      normalizedChildEnd < start ||\n                      childEnd > scanTime\n                    );\n                    \n                    if (!isOnCriticalPath) {\n                      continue;\n                    }\n                    \n                    def childPath = path.clone();\n                    \n                    childPath.add(child.operationId);\n                    \n                    def childId = toHash(childPath);\n                    \n                    if(!childNodes.contains(childId)) {\n                      childNodes.add(childId);\n                    }\n                    \n                    if (normalizedChildEnd < (scanTime - 1000)) {\n                      count(context, nodeId, scanTime - normalizedChildEnd); \n                    }\n                    \n                    scan(context, child, normalizedChildStart, childEnd, childPath);\n                    \n                    scanTime = normalizedChildStart;\n                  }\n                  \n                  if (scanTime > start) {\n                    count(context, nodeId, scanTime - start);\n                  }\n                  \n                }\n              \n                def events = [:];\n                def metadata = [:];\n                def processedEvents = [:];\n                def entryTransactions = [];\n                def timeByNodeId = [:];\n                def nodes = [:];\n                def rootNodes = [];\n                def operationIdByNodeId = [:];\n                \n                \n                def context = [\n                  \"events\": events,\n                  \"metadata\": metadata,\n                  \"processedEvents\": processedEvents,\n                  \"entryTransactions\": entryTransactions,\n                  \"timeByNodeId\": timeByNodeId,\n                  \"nodes\": nodes,\n                  \"operationIdByNodeId\": operationIdByNodeId,\n                  \"params\": params\n                ];\n              \n                for(state in states) {\n                  if (state.eventsById != null) {\n                    events.putAll(state.eventsById);\n                  }\n                  if (state.metadataByOperationId != null) {\n                    metadata.putAll(state.metadataByOperationId);\n                  }\n                }\n                \n                \n                for(def event: events.values()) {\n                  processEvent(context, event);\n                }\n                \n                for(transaction in context.entryTransactions) {\n                  transaction.skew = 0;\n                  transaction.offset = 0;\n                  setOffsetAndSkew(context, transaction, null, transaction.timestamp);\n                  \n                  def path = [];\n                  def parent = transaction;\n                  while (parent != null) {\n                    path.add(parent.operationId);\n                    if (parent.parentId == null) {\n                      break;\n                    }\n                    parent = context.processedEvents[parent.parentId];\n                  }\n\n                  Collections.reverse(path);\n\n                  def nodeId = toHash(path);\n                  \n                  scan(context, transaction, 0, transaction.duration, path);\n                  \n                  if (!rootNodes.contains(nodeId)) {\n                    rootNodes.add(nodeId);\n                  }\n                  \n                }\n                \n                return [\n                  \"timeByNodeId\": timeByNodeId,\n                  \"metadata\": metadata,\n                  \"nodes\": nodes,\n                  \"rootNodes\": rootNodes,\n                  \"operationIdByNodeId\": operationIdByNodeId\n                ];"
                                }
                            }
                        }
                    }
                },
                "name": "get_aggregated_critical_path (/internal/apm/traces/aggregated_critical_path)",
                "response": {
                    "json": {
                        "name": "ResponseError",
                        "message": "{\"ok\":false,\"message\":\"The instance rejected the connection.\"}"
                    }
                },
                "startTime": 1710780063428,
                "stats": {
                    "kibanaApiQueryParameters": {
                        "label": "Kibana API query parameters",
                        "description": "The query parameters used in the Kibana API request that initiated the Elasticsearch request.",
                        "value": "{\n  \"_inspect\": \"true\"\n}"
                    },
                    "kibanaApiRoute": {
                        "label": "Kibana API route",
                        "description": "The route of the Kibana API request that initiated the Elasticsearch request.",
                        "value": "POST /internal/apm/traces/aggregated_critical_path"
                    },
                    "indexPattern": {
                        "label": "Data view",
                        "value": [
                            "traces-apm*",
                            "apm-*"
                        ],
                        "description": "The data view that connected to the Elasticsearch indices."
                    }
                },
                "status": 2
            }
        ]
    }
}

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[chrisdistasio]

I wonder if this is unique to serverless or if the same challenge exists in ESS? Can we determine this?

As a Tech Preview feature, we can prioritize this investigation lower than GA features like Service Map

[dgieselaar]

@chrisdistasio for context, the feature is built on a scripted_metric aggregation which we kind of expect to break down in some cases. The issues are similar to the service map, in the sense that we need to look at trace events and cannot use aggregated metrics, and thus its performance characteristics become unpredictable. We can build in some safe guards to get it to GA though. Happy to help out if needed.

[ablnk]

@chrisdistasio issue is not unique to serverless, just reproduced it stateful deployment too.

[crespocarlos]

Hey @dgieselaar what guardrails you have in mind for this?

[crespocarlos]

This might be related to #181790

[paulb-elastic]

Reiterating the points made above, Traces Explorer is in Tech Preview and this affects both serverless and stateful - as such, this is a lower priority.

[dgieselaar]

@paulb-elastic FWIW, the concerns around service maps were not "this breaks for our users", but "this can take down a cluster and page the ES team and it isn't their responsibility". I think the same applies here, but the risk is lower due the fact it is not enabled by default. I assume the ES team still wants a fix for this as well though.

[neptunian]

@chrisdistasio I think @dgieselaar 's comment above answers your question from earlier about whether or not there is currently a mechanism in elasticsearch to guard itself against being taken down when handling our requests and that we need to build for that on our end.

[crespocarlos]

I can't really think of an alternative other than rewriting the query not to use scripted_metrics aggregation. @dgieselaar @neptunian do you think the investigation conducted here https://github.com/elastic/kibana/issues/179229#issuecomment-2163872017 would help in this scenario too?

Perhaps the fact that it still in technical preview it also gives us more flexibility to rewrite this feature to not use scripted_metrics aggs, provided that there will be performance gains in doing so.

[crespocarlos]

I'm constantly experiencing another issue where, for the same time range, sometimes the data is returned by the server, while other times it keeps loading forever or returns empty.

[paulb-elastic]

As per the discussion with @chrisdistasio, this won't be tackled right now, but moved to the backlog