Error (memory access out of bounds). Please check the server logs for more details.

[mk-raven]

TraceQL query look like: {resource.service.name="Finance.Service" && duration>1s && name=~"FinancePayoutQueueV2 receive|FinancePayoutQueueV2 process"}

Server log: {"level":50,"time":1706875858908,"pid":23,"hostname":"qryn-read-fc4555bcd-8lth5","name":"qryn","reqId":"req-8n0","err":"memory access out of bounds\nRuntimeError: memory access out of bounds\n at createCtx (wasm://wasm/0120487e:wasm-function[933]:0x187996)\n at createCtx.command_export (wasm://wasm/0120487e:wasm-function[966]:0x197557)\n at Ctx.create (/app/wasm_parts/main.js:187:25)\n at module.exports.TranspileTraceQL (/app/wasm_parts/main.js:126:10)\n at search (/app/traceql/index.js:40:15)\n at searchV2 (/app/lib/handlers/tempo_search.js:74:26)\n at Object.handler (/app/lib/handlers/tempo_search.js:25:18)\n at routeOptions.handler (/app/node_modules/@fastify/websocket/index.js:150:24)\n at preHandlerCallback (/app/node_modules/fastify/lib/handleRequest.js:137:37)\n at validationCompleted (/app/node_modules/fastify/lib/handleRequest.js:121:5)","msg":"memory access out of bounds"}

[deathalt]

cannot reproduce in qryn 3.2.2

[mk-raven]

I think its fixed with new update of qryn version.

[akvlad]

I'm not sure. Let's keep this open. Please notify if you observer the error again.

[lmangani]

@mk-raven could you confirm if the situation looks stable with the latest version?

[akvlad]

@lmangani I have spared the issue for the future investigations. So for now it likely has no complaints.

[shimaore]

Not sure if this is the same error, but seeing this intermittently with 3.2.10:

{"level":30,"time":1710154424815,"pid":14,"name":"qryn","reqId":"req-g","req":{"method":"GET","url":"/api/v1/query_range?end=1710154410&query=voicekamailio_database_failures_response_total&start=1710150810&step=15","hostname":"127.0.0.1:3099","remoteAddress":"127.0.0.1","remotePort":34116},"msg":"incoming request"}
Empty <[Object: null prototype] {}> {
  end: '1710154410',
  query: 'voicekamailio_database_failures_response_total',
  start: '1710150810',
  step: '15'
}
RuntimeError: memory access out of bounds
    at createCtx (wasm://wasm/0120487e:wasm-function[933]:0x187996)
    at createCtx.command_export (wasm://wasm/0120487e:wasm-function[966]:0x197557)
    at Ctx.create (/app/wasm_parts/main.js:187:25)
    at pql (/app/wasm_parts/main.js:151:9)
    at module.exports.pqlRangeQuery (/app/wasm_parts/main.js:59:16)
    at module.exports.rangeQuery (/app/promql/index.js:20:35)
    at Object.handler (/app/lib/handlers/prom_query_range.js:22:26)
    at routeOptions.handler (/app/node_modules/@fastify/websocket/index.js:150:24)
    at preHandlerCallback (/app/node_modules/fastify/lib/handleRequest.js:137:37)
    at validationCompleted (/app/node_modules/fastify/lib/handleRequest.js:121:5)
{"level":30,"time":1710154424829,"pid":14,"name":"qryn","reqId":"req-g","res":{"statusCode":500},"responseTime":14.050364017486572,"msg":"request completed"}

[akvlad]

@shimaore @mk-raven @deathalt Good news. A big complex update on the TraceQL and Metrics engines have been released in v3.2.11. Should help with the memory access out of bounds error and improve the performance of metrics reading.

[shimaore]

@akvlad updated to v3.2.11, still getting an out-of-bound error intermittently (see below). This seems to be in a different location though.

{"level":30,"time":1710189653278,"pid":14,"name":"qryn","reqId":"req-g","req":{"method":"GET","url":"/api/v1/query_range?end=1710189645&query=rate%28node_cpu_seconds_total%7Bmode%3D%22iowait%22%7D%5B1m0s%5D%29&start=1710186045&step=15","hostname":"127.0.0.1:3099","remoteAddress":"127.0.0.1","remotePort":58168},"msg":"incoming request"}
Empty <[Object: null prototype] {}> {
  end: '1710189645',
  query: 'rate(node_cpu_seconds_total{mode="iowait"}[1m0s])',
  start: '1710186045',
  step: '15'
}
RuntimeError: memory access out of bounds
    at (*wasm_parts/regexp.Regexp).tryBacktrack (wasm://wasm/00d8aaf6:wasm-function[938]:0xd5b27)
    at (*wasm_parts/regexp.Regexp).doExecute (wasm://wasm/00d8aaf6:wasm-function[936]:0xd49b8)
    at github.com/prometheus/common/model.ParseDuration (wasm://wasm/00d8aaf6:wasm-function[700]:0x907a6)
    at (*github.com/prometheus/prometheus/promql/parser.yyParserImpl).Parse (wasm://wasm/00d8aaf6:wasm-function[985]:0xe16df)
    at github.com/prometheus/prometheus/promql/parser.ParseExpr (wasm://wasm/00d8aaf6:wasm-function[1026]:0xebc4e)
    at (*github.com/prometheus/prometheus/promql.Engine).NewRangeQuery (wasm://wasm/00d8aaf6:wasm-function[1185]:0x112737)
    at main.pqlRangeQuery$1 (wasm://wasm/00d8aaf6:wasm-function[1358]:0x150a39)
    at main.pql (wasm://wasm/00d8aaf6:wasm-function[1359]:0x150cac)
    at pqlRangeQuery (wasm://wasm/00d8aaf6:wasm-function[1357]:0x150597)
    at pqlRangeQuery.command_export (wasm://wasm/00d8aaf6:wasm-function[1390]:0x157c68)
{"level":30,"time":1710189653279,"pid":14,"name":"qryn","reqId":"req-g","res":{"statusCode":500},"responseTime":0.8004550933837891,"msg":"request completed"}

[akvlad]

@shimaore apparently some changes were not comitted. Should be fixed in v3.2.12. Sorry. Thanks for the quick response.

[shimaore]

@akvlad no issue, thank you for the quick response as well!

Not seeing the out-of-bound errors anymore on my setup with v3.2.12 👍

[lmangani]

Closing as completed! Feel free to reopen if the issue persists