search

openshift / cluster-debug-tools

Unsupported tools used to investigate broken clusters.
Apache License 2.0
26 stars 28 forks source link

POC:WIP: detect increased requests latency from audit logs. #48

Open p0lyn0mial opened 2 months ago

p0lyn0mial commented 2 months ago 
❯ ./kubectl-dev_tool audit -f audit.log --stage=ResponseComplete -o stats

increased latency for tracker: apiserver.latency.k8s.io/etcd detected, from: 2024-07-23 01:26:15.374266 +0200 CEST, to: 2024-07-23 01:28:08.084897 +0200 CEST, totalEvents: 4707

tracker=apiserver.latency.k8s.io/etcd, min=5.999151357s max=1m0.003963671s median=14.999281279s 90th=59.999766772s events=1432 [2024-07-23 01:26:15.374266 +0200 CEST - 2024-07-23 01:27:11.864862 +0200 CEST]

tracker=apiserver.latency.k8s.io/etcd, min=5.999662741s max=1m0.003140176s median=14.999079115s 90th=59.99946593s events=1368 [2024-07-23 01:26:41.953009 +0200 CEST - 2024-07-23 01:27:41.865684 +0200 CEST]

tracker=apiserver.latency.k8s.io/etcd, min=5.999838559s max=1m0.000907557s median=14.998924886s 90th=56.156543455s events=1304 [2024-07-23 01:27:11.952965 +0200 CEST - 2024-07-23 01:28:08.084897 +0200 CEST]

tracker=apiserver.latency.k8s.io/etcd, min=6.000189594s max=1m0.000907557s median=14.998449838s 90th=21.002733985s events=603 [2024-07-23 01:27:41.95378 +0200 CEST - 2024-07-23 01:28:08.084897 +0200 CEST]

increased latency for tracker: apiserver.latency.k8s.io/etcd detected, from: 2024-07-23 01:32:30.189283 +0200 CEST, to: 2024-07-23 01:34:18.424754 +0200 CEST, totalEvents: 3362

tracker=apiserver.latency.k8s.io/etcd, min=5.999939831s max=1m0.000961944s median=15.000243142s 90th=59.99997495s events=758 [2024-07-23 01:32:30.189283 +0200 CEST - 2024-07-23 01:33:11.864284 +0200 CEST]

tracker=apiserver.latency.k8s.io/etcd, min=5.999750971s max=1m0.000961944s median=14.999699291s 90th=59.999637897s events=966 [2024-07-23 01:32:41.952189 +0200 CEST - 2024-07-23 01:33:41.867796 +0200 CEST]

tracker=apiserver.latency.k8s.io/etcd, min=4.158830543s max=1m0.000746847s median=14.99908652s 90th=59.482115474s events=975 [2024-07-23 01:33:11.953796 +0200 CEST - 2024-07-23 01:34:11.867509 +0200 CEST]

tracker=apiserver.latency.k8s.io/etcd, min=4.158830543s max=1m0.000686771s median=14.99870116s 90th=15.023435927s events=564 [2024-07-23 01:33:41.952985 +0200 CEST - 2024-07-23 01:34:18.424754 +0200 CEST]

tracker=apiserver.latency.k8s.io/etcd, min=5.999903894s max=59.999955725s median=14.999043598s 90th=15.000539502s events=99 [2024-07-23 01:34:11.952101 +0200 CEST - 2024-07-23 01:34:18.424754 +0200 CEST]

increased latency for tracker: apiserver.latency.k8s.io/etcd detected, from: 2024-07-23 01:34:11.952101 +0200 CEST, to: 2024-07-23 01:34:18.424754 +0200 CEST, totalEvents: 99

tracker=apiserver.latency.k8s.io/etcd, min=5.999903894s max=59.999955725s median=14.999043598s 90th=15.000539502s events=99 [2024-07-23 01:34:11.952101 +0200 CEST - 2024-07-23 01:34:18.424754 +0200 CEST]
openshift-merge-robot commented 1 month ago

PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.