metrico / qryn

Polyglot Observability Stack. Lightweight & Drop-in compatible with Loki, Prometheus, Tempo, Pyroscope, Opentelemetry, Datadog & more! WASM powered ⭐️ Star to Support
https://qryn.dev
GNU Affero General Public License v3.0
1.05k stars 63 forks source link

fix: OOM in traceql requests #502

Closed akvlad closed 1 month ago

akvlad commented 2 months ago

traceql request on more than 10M traces may cause OOM. The PR should fix the OOM

github-actions[bot] commented 1 month ago

Workflow Telemetry - K6 Test qxip/qryn / Run k6 on qryn:latest

Workflow telemetry for commit 22358f48d0684c899ec664712a813195444152a4 You can access workflow job details here

Step Trace

gantt
    title Run k6 on qryn:latest
    dateFormat x
    axisFormat %H:%M:%S
    Set up job : milestone, 1716218580000, 1716218582000
    Initialize containers : 1716218582000, 1716218601000
    Checkout : 1716218601000, 1716218603000
    Run actions/setup-go@v4 : 1716218603000, 1716218605000
    Download k6 for Prometheus : 1716218605000, 1716218606000
    Download k6 for Prometheus : 1716218606000, 1716218606000
    Collect Workflow Telemetry : 1716218606000, 1716218606000
    Wait for Loki API : 1716218606000, 1716218606000
    Run k6 loki test : 1716218606000, 1716218670000
    Run mv summary.txt loki_summary.txt : 1716218670000, 1716218670000
    Wait for Prom API : 1716218670000, 1716218670000
    Run k6 prometheus test : 1716218670000, 1716218731000
    Run mv summary.txt prom_summary.txt : 1716218731000, 1716218731000
    Wait for Tempo API : 1716218731000, 1716218731000
    Run k6 tempo test : 1716218731000, 1716218791000
    Run mv summary.txt tempo_summary.txt : 1716218791000, 1716218791000
    Strip Text report : 1716218791000, 1716218791000
    Run actions/upload-artifact@v3 : 1716218791000, 1716218791000
    Summary Report : 1716218791000, 1716218791000

CPU Metrics

chart_stacked_area_time_9b820606-4d50-4298-bfe7-0510df1b4c42

Memory Metrics

chart_stacked_area_time_7434529c-c4b0-4f8d-96c4-98aa7e1678f4

IO Metrics

Read Write
Network I/O chart_line_time_70355311-9761-478b-9f38-38c86c9d050c chart_line_time_9726b462-a89d-4388-8d68-704b65d4b998
Disk I/O chart_line_time_ffaa0c5f-70d9-409c-9d4b-51345c07b2ef chart_line_time_82f7bc3d-2269-43f8-a55b-1e037517db24

Process Trace

Top 100 processes with highest duration

gantt
    title Run k6 on qryn:latest
    dateFormat x
    axisFormat %H:%M:%S
github-actions[bot] commented 1 month ago

Workflow Telemetry - K6 Test qxip/qryn / Run k6 on qryn:latest

Workflow telemetry for commit aa2babc191107d50aa80799d4b2d2b184f1f4941 You can access workflow job details here

Step Trace

gantt
    title Run k6 on qryn:latest
    dateFormat x
    axisFormat %H:%M:%S
    Set up job : milestone, 1716456030000, 1716456033000
    Initialize containers : 1716456033000, 1716456052000
    Checkout : 1716456052000, 1716456054000
    Run actions/setup-go@v4 : 1716456054000, 1716456054000
    Download k6 for Prometheus : 1716456054000, 1716456055000
    Download k6 for Prometheus : 1716456055000, 1716456055000
    Collect Workflow Telemetry : 1716456055000, 1716456056000
    Wait for Loki API : 1716456056000, 1716456056000
    Run k6 loki test : 1716456056000, 1716456120000
    Run mv summary.txt loki_summary.txt : 1716456120000, 1716456120000
    Wait for Prom API : 1716456120000, 1716456120000
    Run k6 prometheus test : 1716456120000, 1716456182000
    Run mv summary.txt prom_summary.txt : 1716456182000, 1716456182000
    Wait for Tempo API : 1716456182000, 1716456182000
    Run k6 tempo test : 1716456182000, 1716456241000
    Run mv summary.txt tempo_summary.txt : 1716456241000, 1716456241000
    Strip Text report : 1716456241000, 1716456241000
    Run actions/upload-artifact@v3 : 1716456241000, 1716456243000
    Summary Report : 1716456243000, 1716456243000

CPU Metrics

chart_stacked_area_time_0c8b709e-1fa6-41c1-bfa2-7071d5862ebc

Memory Metrics

chart_stacked_area_time_7097d1ef-c811-4eb9-9854-1df64ec7f254

IO Metrics

Read Write
Network I/O chart_line_time_03a6058d-6266-4dd3-b491-a42f987f7ae5 chart_line_time_16a2f98b-d449-4ffd-ab74-e3c79e1adee8
Disk I/O chart_line_time_f7a3830b-9dff-45b7-a9f3-f44b2d3a7258 chart_line_time_f76e0cd4-cae2-46df-a947-6888b0dbc3c1

Process Trace

Top 100 processes with highest duration

gantt
    title Run k6 on qryn:latest
    dateFormat x
    axisFormat %H:%M:%S