search

tbg / cockroach

A Scalable, Geo-Replicated, Transactional Datastore
Other
1 stars 0 forks source link

Elevated latency #181

Open tbg opened 1 year ago

tbg commented 1 year ago

Customer: Fakely, Inc Deployment: Cloud (really roachprod but let's pretend) Version: 22.2.0

One of our application teams has alerted us to recurring periods of elevated latency (>50ms p99) which cause them to miss their SLAs. We need assistance tracking down the source of the problem.

image

Grafana

http://35.231.99.87:3000/d/xJ2AWtF4z/new-dashboard?orgId=1&refresh=5s&from=now-15m&to=now

Admin UIs

http://35.185.112.74:26258/ http://34.139.230.17:26258/ http://34.148.94.36:26258/ http://35.227.42.232:26258/ http://34.148.61.172:26258/ http://35.231.99.87:26258/

tbg commented 1 year ago

This is roachprod, export GCE_PROJECT=andrei-jepsen

debug.zip: roachprod get tobias-p99:1 debug.zip