search

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.89k stars 9.78k forks source link

Etcd cluster keeps getting corrupted on Talos since version 1.8.x upgrade ( etcd 3.5.16) #18933

Open rkoosaar opened 19 hours ago

rkoosaar commented 19 hours ago

Bug report criteria

What happened?

Hi folks, I have a really odd issue that I'm troubleshooting. I have a 3 node Talos (1.8.3) cluster at home where etcd (3.5.16) keeps getting corrupted after a while. Initially I thought it could be a disk related issue. So I bought brand new disks and swapped them around. I installed a new cluster last night (around 8pm) and when I woke up this morning (8am) cluster was not working and etcd was reporting cluster corrupted. Looking at the logs, it seems something happened around 6am, but I'm unable to work out what the cause is. So far I have redeployed the cluster in the past week 4 times and every time etcd has ended up corrupted. Any help/guidance to troubleshoot this would be much appreciated.

What did you expect to happen?

Cluster not the get corrupted

How can we reproduce it (as minimally and precisely as possible)?

I'm not 100% sure how this can be reproduced in your env as I don't fully understand why this happens

Anything else we need to know?

I have actually saved a log bundles from all 3 cluster nodes using talosctl -n node_ip support I'm just not sure which log files would be helpful. If you could advise which logs are needed I can provide them: the log bundle has folders: kubernetes-logs service-logs (etcd.log file here, I pasted it in the relevant log section) and separately log files: controller-runtime.log dmesg.log

Etcd version (please run commands below)

here is the output of EtcdConfigs.etcd.talos.dev file from node1

```console $ etcd --version # paste output here metadata: namespace: etcd type: EtcdConfigs.etcd.talos.dev id: etcd version: 1 owner: etcd.ConfigController phase: running created: 2024-11-20T20:09:34Z updated: 2024-11-20T20:09:34Z spec: advertiseValidSubnets: - 10.1.1.0/24 advertiseExcludeSubnets: - 10.1.1.30 listenValidSubnets: - 10.1.1.0/24 listenExcludeSubnets: [] image: gcr.io/etcd-development/etcd:v3.5.16 extraArgs: listen-metrics-urls: http://0.0.0.0:2381 $ etcdctl version # paste output here ```

Etcd configuration (command line flags or environment variables)

# paste your configuration here metadata: namespace: etcd type: EtcdConfigs.etcd.talos.dev id: etcd version: 1 owner: etcd.ConfigController phase: running created: 2024-11-20T20:09:34Z updated: 2024-11-20T20:09:34Z spec: advertiseValidSubnets: - 10.1.1.0/24 advertiseExcludeSubnets: - 10.1.1.30 listenValidSubnets: - 10.1.1.0/24 listenExcludeSubnets: [] image: gcr.io/etcd-development/etcd:v3.5.16 extraArgs: listen-metrics-urls: http://0.0.0.0:2381

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

I'm not 100% sure how I can run the below commands on talos

```console $ etcdctl member list -w table # paste output here talosctl -n 10.1.1.32 etcd members NODE ID HOSTNAME PEER URLS CLIENT URLS LEARNER 10.1.1.32 05f2a341182982bc nectarine https://10.1.1.33:2380 https://10.1.1.33:2379 false 10.1.1.32 4ce3ddbc6cc0469b melon https://10.1.1.32:2380 https://10.1.1.32:2379 false 10.1.1.32 d9983876898d0723 mango https://10.1.1.31:2380 https://10.1.1.31:2379 false $ etcdctl --endpoints= endpoint status -w table # paste output here ```

Relevant log output

id:0723934b321f01b0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708327Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.25µs","request":"header:<ID:5087822155244138456 > lease_revoke:<id:02bc934b32ef139b>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708355Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef139b","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708372Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321f01b0","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709189Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.598µs","request":"header:<ID:5087822155244138455 > lease_revoke:<id:0723934b321e383c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709267Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.711µs","request":"header:<ID:5087822155244138457 > lease_revoke:<id:02bc934b32ef5746>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709318Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef5746","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709336Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321e383c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.840207Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.109µs","request":"header:<ID:9420566471738703565 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/network/ingress-nginx-internal-leader\" mod_revision:482768 > success:<request_put:<key:\"/registry/leases/network/ingress-nginx-internal-leader\" value_size:439 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.846801Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.293µs","request":"header:<ID:514416733623534472 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/spegel-leader-election\" mod_revision:482773 > success:<request_put:<key:\"/registry/leases/kube-system/spegel-leader-election\" value_size:450 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.922691Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.287µs","request":"header:<ID:5087822155244138460 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" mod_revision:482782 > success:<request_put:<key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" value_size:542 >> failure:<request_range:<key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.969917Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.089µs","request":"header:<ID:514416733623534475 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/spegel-leader-election\" mod_revision:482773 > success:<request_put:<key:\"/registry/leases/kube-system/spegel-leader-election\" value_size:450 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.988979Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.809µs","request":"header:<ID:514416733623534478 username:\"kube-apiserver\" auth_revision:1 > lease_grant:<ttl:15-second id:0723934b3225c38d>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.039835Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.076µs","request":"header:<ID:5087822155244138461 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" mod_revision:474638 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" value_size:8401 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207388Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.406µs","request":"header:<ID:5087822155244138465 > lease_revoke:<id:0723934b321bc809>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207492Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.201µs","request":"header:<ID:5087822155244138466 > lease_revoke:<id:02bc934b32ef3ecb>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207531Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3ecb","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207598Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bc809","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208243Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.962µs","request":"header:<ID:5087822155244138468 > lease_revoke:<id:0723934b321e790a>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208313Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.534µs","request":"header:<ID:5087822155244138467 > lease_revoke:<id:02bc934b32ef3db5>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208366Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"20.283µs","request":"header:<ID:5087822155244138469 > lease_revoke:<id:0723934b321c69c9>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208351Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321e790a","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208410Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.463µs","request":"header:<ID:5087822155244138470 > lease_revoke:<id:02bc934b32ef09ab>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208424Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3db5","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208448Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321c69c9","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208448Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef09ab","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.218575Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.363µs","request":"header:<ID:514416733623534481 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:412 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.246152Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.879µs","request":"header:<ID:9420566471738703566 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.254819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.528µs","request":"header:<ID:9420566471738703567 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.263370Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.398µs","request":"header:<ID:9420566471738703568 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.275016Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.819µs","request":"header:<ID:9420566471738703569 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.283479Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.384µs","request":"header:<ID:9420566471738703570 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.296399Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.681µs","request":"header:<ID:9420566471738703572 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.304905Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.013µs","request":"header:<ID:9420566471738703573 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.314273Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.867µs","request":"header:<ID:9420566471738703574 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.322761Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.973µs","request":"header:<ID:9420566471738703575 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.332116Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.185µs","request":"header:<ID:9420566471738703576 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.472883Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.726µs","request":"header:<ID:9420566471738703578 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/database/db9c8771.cnpg.io\" mod_revision:482784 > success:<request_put:<key:\"/registry/leases/database/db9c8771.cnpg.io\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.478827Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.75µs","request":"header:<ID:5087822155244138473 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" mod_revision:456502 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" value_size:7837 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.498891Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.983µs","request":"header:<ID:514416733623534485 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/helm-controller-64cd59b777-l22bw\" mod_revision:2901 > success:<request_put:<key:\"/registry/pods/flux-system/helm-controller-64cd59b777-l22bw\" value_size:5391 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.520308Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.01µs","request":"header:<ID:514416733623534487 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/openebs-system/openebs-localpv-provisioner-56d8447b9-kbxm7\" mod_revision:4743 > success:<request_put:<key:\"/registry/pods/openebs-system/openebs-localpv-provisioner-56d8447b9-kbxm7\" value_size:4789 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.544561Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.851µs","request":"header:<ID:514416733623534491 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:482767 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:432 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.549218Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.825µs","request":"header:<ID:514416733623534492 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/kustomize-controller-54f65db458-fg22s\" mod_revision:3124 > success:<request_put:<key:\"/registry/pods/flux-system/kustomize-controller-54f65db458-fg22s\" value_size:5343 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.582441Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.94µs","request":"header:<ID:5087822155244138474 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" mod_revision:474686 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" value_size:8387 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.708052Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.388µs","request":"header:<ID:5087822155244138475 > lease_revoke:<id:02bc934b32ef1d03>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.708127Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef1d03","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709060Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.758µs","request":"header:<ID:5087822155244138476 > lease_revoke:<id:0723934b321deb0c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709134Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.642µs","request":"header:<ID:5087822155244138478 > lease_revoke:<id:02bc934b32ef3bcf>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709165Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.181µs","request":"header:<ID:5087822155244138477 > lease_revoke:<id:469b934b33aa80b1>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709191Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.744µs","request":"header:<ID:5087822155244138479 > lease_revoke:<id:02bc934b32ef1fa0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709214Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.752µs","request":"header:<ID:5087822155244138480 > lease_revoke:<id:0723934b321d2570>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709222Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321deb0c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709286Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3bcf","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709313Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef1fa0","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709293Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d2570","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709333Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aa80b1","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.852623Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.431µs","request":"header:<ID:514416733623534495 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:410 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.993053Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.448µs","request":"header:<ID:9420566471738703580 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/external-resizer-driver-longhorn-io\" mod_revision:482772 > success:<request_put:<key:\"/registry/leases/storage/external-resizer-driver-longhorn-io\" value_size:410 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.028156Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.859µs","request":"header:<ID:9420566471738703582 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/driver-longhorn-io\" mod_revision:482778 > success:<request_put:<key:\"/registry/leases/storage/driver-longhorn-io\" value_size:404 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.028822Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.164µs","request":"header:<ID:5087822155244138481 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" mod_revision:361515 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" value_size:7850 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.044577Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.882µs","request":"header:<ID:5087822155244138482 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" mod_revision:475880 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" value_size:8399 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.044630Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.259µs","request":"header:<ID:5087822155244138483 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" mod_revision:481730 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" value_size:8400 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.045521Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.235µs","request":"header:<ID:5087822155244138484 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" mod_revision:482731 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" value_size:8401 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.056059Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.953µs","request":"header:<ID:9420566471738703585 username:\"kube-apiserver\" auth_revision:1 > lease_grant:<ttl:15-second id:02bc934b32f0bee0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.139893Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.315µs","request":"header:<ID:514416733623534498 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kubelet-csr-approver\" mod_revision:482781 > success:<request_put:<key:\"/registry/leases/kube-system/kubelet-csr-approver\" value_size:451 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.187110Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.253µs","request":"header:<ID:514416733623534500 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/source-controller-leader-election\" mod_revision:482775 > success:<request_put:<key:\"/registry/leases/flux-system/source-controller-leader-election\" value_size:458 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.195634Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.015µs","request":"header:<ID:514416733623534502 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/kustomize-controller-leader-election\" mod_revision:482740 > success:<request_put:<key:\"/registry/leases/flux-system/kustomize-controller-leader-election\" value_size:468 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.208502Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.898µs","request":"header:<ID:5087822155244138486 > lease_revoke:<id:0723934b321bfe07>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.208593Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bfe07","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209357Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.243µs","request":"header:<ID:5087822155244138487 > lease_revoke:<id:0723934b321eb240>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209427Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.749µs","request":"header:<ID:5087822155244138488 > lease_revoke:<id:02bc934b32ef52fd>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209463Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.023µs","request":"header:<ID:5087822155244138489 > lease_revoke:<id:0723934b321ec88f>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209488Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.756µs","request":"header:<ID:5087822155244138490 > lease_revoke:<id:0723934b321dfb59>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209513Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.696µs","request":"header:<ID:5087822155244138492 > lease_revoke:<id:02bc934b32ef453e>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209517Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321eb240","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209544Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.952µs","request":"header:<ID:5087822155244138491 > lease_revoke:<id:02bc934b32ef2ba3>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209565Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef453e","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209592Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321dfb59","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209588Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321ec88f","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209628Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef2ba3","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209631Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef52fd","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.477676Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.433µs","request":"header:<ID:5087822155244138493 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" mod_revision:482626 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" value_size:8390 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.494209Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.532µs","request":"header:<ID:514416733623534506 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/image-reflector-controller-leader-election\" mod_revision:482755 > success:<request_put:<key:\"/registry/leases/flux-system/image-reflector-controller-leader-election\" value_size:488 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710090Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.661µs","request":"header:<ID:5087822155244138494 > lease_revoke:<id:469b934b33aa516c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710191Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aa516c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710658Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.44µs","request":"header:<ID:5087822155244138495 > lease_revoke:<id:02bc934b32ef4e33>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710740Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.006µs","request":"header:<ID:5087822155244138497 > lease_revoke:<id:02bc934b32ef5148>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710799Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef4e33","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.853µs","request":"header:<ID:5087822155244138496 > lease_revoke:<id:0723934b321c30b9>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710847Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef5148","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710896Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.145µs","request":"header:<ID:5087822155244138498 > lease_revoke:<id:469b934b33aab5f6>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710938Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.95µs","request":"header:<ID:5087822155244138499 > lease_revoke:<id:0723934b321d9310>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710978Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.44µs","request":"header:<ID:5087822155244138500 > lease_revoke:<id:0723934b321d6665>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710963Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321c30b9","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710978Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aab5f6","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.711009Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d6665","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.711018Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d9310","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.796843Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.747µs","request":"header:<ID:514416733623534509 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/cilium-operator-resource-lock\" mod_revision:482771 > success:<request_put:<key:\"/registry/leases/kube-system/cilium-operator-resource-lock\" value_size:408 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.837363Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.5µs","request":"header:<ID:514416733623534511 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/external-snapshotter-leader-driver-longhorn-io\" mod_revision:482745 > success:<request_put:<key:\"/registry/leases/storage/external-snapshotter-leader-driver-longhorn-io\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.897227Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.009µs","request":"header:<ID:514416733623534513 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:482767 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.912839Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.143µs","request":"header:<ID:514416733623534515 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/storage/csi-attacher-787d99954-j6tt7\" mod_revision:12987 > success:<request_put:<key:\"/registry/pods/storage/csi-attacher-787d99954-j6tt7\" value_size:3913 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.934714Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.311µs","request":"header:<ID:514416733623534517 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/cert-manager/cert-manager-cainjector-c879b86d6-w7tg9\" mod_revision:130011 > success:<request_put:<key:\"/registry/pods/cert-manager/cert-manager-cainjector-c879b86d6-w7tg9\" value_size:4165 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.957608Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.483µs","request":"header:<ID:514416733623534519 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/security/authentik-worker-6c9b654886-bj8vj\" mod_revision:478106 > success:<request_put:<key:\"/registry/pods/security/authentik-worker-6c9b654886-bj8vj\" value_size:4627 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.982359Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.64µs","request":"header:<ID:514416733623534521 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/notification-controller-556b8867f8-79nbc\" mod_revision:2849 > success:<request_put:<key:\"/registry/pods/flux-system/notification-controller-556b8867f8-79nbc\" value_size:5424 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.000088Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.055µs","request":"header:<ID:514416733623534523 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:475 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.009692Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.335µs","request":"header:<ID:514416733623534524 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/storage/volsync-8466895bb6-mmlrf\" mod_revision:12652 > success:<request_put:<key:\"/registry/pods/storage/volsync-8466895bb6-mmlrf\" value_size:6091 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.011609Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.855µs","request":"header:<ID:514416733623534525 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:473 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.019020Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.332µs","request":"header:<ID:5087822155244138501 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" mod_revision:480448 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" value_size:8399 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.020536Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.689µs","request":"header:<ID:514416733623534526 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.029988Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"11.455µs","request":"header:<ID:514416733623534528 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.038232Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.985µs","request":"header:<ID:514416733623534529 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/database/cloudnative-pg-586475f9d5-drx9k\" mod_revision:29569 > success:<request_put:<key:\"/registry/pods/database/cloudnative-pg-586475f9d5-drx9k\" value_size:5602 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.040470Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.149µs","request":"header:<ID:514416733623534530 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.043611Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"14.402µs","request":"header:<ID:5087822155244138502 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" mod_revision:482081 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" value_size:7851 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.053946Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.008µs","request":"header:<ID:514416733623534533 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.059050Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.624µs","request":"header:<ID:514416733623534534 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/cert-manager/cert-manager-659dbd9c7f-cmm4j\" mod_revision:130005 > success:<request_put:<key:\"/registry/pods/cert-manager/cert-manager-659dbd9c7f-cmm4j\" value_size:4795 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.064973Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.353µs","request":"header:<ID:514416733623534535 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.067134Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.292µs","request":"header:<ID:9420566471738703587 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/driver-longhorn-io\" mod_revision:482778 > success:<request_put:<key:\"/registry/leases/storage/driver-longhorn-io\" value_size:404 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.073474Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.807µs","request":"header:<ID:514416733623534538 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.076837Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.029µs","request":"header:<ID:514416733623534539 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:408 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.083302Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.122µs","request":"header:<ID:514416733623534540 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/image-reflector-controller-7cdc88c5c7-vjdng\" mod_revision:2937 > success:<request_put:<key:\"/registry/pods/flux-system/image-reflector-controller-7cdc88c5c7-vjdng\" value_size:5448 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.083940Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.07µs","request":"header:<ID:514416733623534541 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.091384Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6µs","request":"header:<ID:514416733623534542 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.210758Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.282µs","request":"header:<ID:5087822155244138506 > lease_revoke:<id:0723934b321bb66b>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.210836Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bb66b","error":"etcdserver: corrupt cluster"}
serathius commented 18 hours ago

Logs you provided are just from request failures due to past corruption alarm. Can you provide the logs from time the corruption happen?

rkoosaar commented 18 hours ago

This is what the log bundle had from the time I generated it. Not sure what happened then or why it starts from that time. Maybe it got overwritten since its constantly saying cluster corrupted. do you think any other log file from the bundle might help? or me uploading the whole bundle?

ahrtr commented 1 hour ago

This is what the log bundle had from the time I generated it.

Could you provide the complete log of all etcd instances?

or me uploading the whole bundle?

Yes, please. Is it possible to upload all the db files (under the ${data_dir}/member/snap/db) If it doesn't have any sensitive data?