search

Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

Enable query tracing & limits in Query Node #5088

Open kdembler opened 6 months ago

kdembler commented 6 months ago

Recently we've had a lot of issues with our QN instances being unavailable or very slow to respond. This is most likely caused by some expensive queries running for a long time basically blocking all the other queries. To fix this situation I think we should do 2 things:

  1. Add ability to track queries that took more than X seconds to execute - query body, request headers so we can see what apps are making those. Ideally we would save that to some log file so it's easy to inspect later.
  2. Ability to abort queries that take longer than X seconds
thesan commented 6 months ago

@kdembler @mnaamani

I addressed (2) with https://github.com/Joystream/hydra/pull/538

About (1) as I wrote on Discord the current open telemetry implementation should already enough that. Can the prod QN instance be run with open telemetry on ? should I change the command to also run it with pm2 ?