Open veenadong opened 2 months ago
Update: Recovered api-rest by restarting the pod. This got the diskpools back. However, io-engine continues to log:
[2024-08-13T18:57:26.262907584+00:00 INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:324] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' connected to subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T18:57:30.982486681+00:00 INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:344] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' disconnected from subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T18:59:33.849688174+00:00 INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:324] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' connected to subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T18:59:37.633400695+00:00 INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:344] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' disconnected from subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T19:00:04.647449691+00:00 INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:324] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' connected to subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T19:01:42.336838311+00:00 INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:344] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' disconnected from subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T19:03:45.060224862+00:00 INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:324] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' connected to subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T19:03:48.878250758+00:00 INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:344] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' disconnected from subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
The pod trying to use this volume:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 3m49s (x361 over 12h) kubelet MountVolume.MountDevice failed for volume "pvc-3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6" : rpc error: code = Internal desc = Failed to stage volume 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6: failed to mount device /dev/nvme38n1 onto /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/c7aaf5d73dfe8640ea445a161e8d9c1cac2fc491de65127619284804b3642647/globalmount: Input/output error (os error 5)
Another pod is logging:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 11s (x234 over 12h) kubelet MountVolume.MountDevice failed for volume "pvc-1b693021-bce1-4490-9916-2a44c3e02a05" : rpc error: code = Unavailable desc = Failed to stage volume 1b693021-bce1-4490-9916-2a44c3e02a05: device attach timeout
While the volume is showing as online:
core@sc-os-165-node3:~$ kubectl mayastor get volume 1b693021-bce1-4490-9916-2a44c3e02a05
ID REPLICAS TARGET-NODE ACCESSIBILITY STATUS SIZE THIN-PROVISIONED ALLOCATED SNAPSHOTS SOURCE
1b693021-bce1-4490-9916-2a44c3e02a05 2 sc-os-165-node3.glcpdev.cloud.hpe.com nvmf Online 10GiB true 1.4GiB 0 <none>
core@sc-os-165-node3:~$ kubectl mayastor get volume 1b693021-bce1-4490-9916-2a44c3e02a05 -o yaml
spec:
num_replicas: 2
size: 10737418240
status: Created
target:
node: sc-os-165-node3.glcpdev.cloud.hpe.com
protocol: nvmf
frontend_nodes:
- name: sc-os-165-node3.glcpdev.cloud.hpe.com
nqn: nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com
uuid: 1b693021-bce1-4490-9916-2a44c3e02a05
topology:
pool_topology: !labelled
exclusion: {}
inclusion:
openebs.io/created-by: operator-diskpool
affinitykey: []
policy:
self_heal: true
thin: true
num_snapshots: 0
state:
target:
children:
- state: Online
uri: bdev:///d89ffaf8-868b-418d-9068-68fd5df377b1?uuid=d89ffaf8-868b-418d-9068-68fd5df377b1
- state: Online
uri: nvmf://10.245.250.167:8420/nqn.2019-05.io.openebs:29e303ed-91f3-4888-bd9f-cf11186fe3a7?uuid=29e303ed-91f3-4888-bd9f-cf11186fe3a7
deviceUri: nvmf://10.245.250.168:8420/nqn.2019-05.io.openebs:1b693021-bce1-4490-9916-2a44c3e02a05?hostnqn=nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com
node: sc-os-165-node3.glcpdev.cloud.hpe.com
rebuilds: 0
protocol: nvmf
size: 10737418240
state: Online
uuid: 6bde92b3-a0e0-4194-a797-0201ec250168
size: 10737418240
status: Online
uuid: 1b693021-bce1-4490-9916-2a44c3e02a05
replica_topology:
d89ffaf8-868b-418d-9068-68fd5df377b1:
node: sc-os-165-node3.glcpdev.cloud.hpe.com
pool: sc-os-165-node3.glcpdev.cloud.hpe.com-disk
state: Online
child-status: Online
usage:
capacity: 10737418240
allocated: 1472200704
allocated_snapshots: 0
allocated_all_snapshots: 0
29e303ed-91f3-4888-bd9f-cf11186fe3a7:
node: sc-os-165-node2.glcpdev.cloud.hpe.com
pool: sc-os-165-node2.glcpdev.cloud.hpe.com-disk
state: Online
child-status: Online
usage:
capacity: 10737418240
allocated: 1472200704
allocated_snapshots: 0
allocated_all_snapshots: 0
usage:
capacity: 10737418240
allocated: 1472200704
allocated_replica: 1472200704
allocated_snapshots: 0
allocated_all_snapshots: 0
total_allocated: 2944401408
total_allocated_replicas: 2944401408
total_allocated_snapshots: 0
Noticed errors in csi-node:
2024-08-13T19:54:21.383875Z ERROR csi_node::filesystem_vol: Failed to stage volume 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6: failed to mount device /dev/nvme38n1 onto /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/c7aaf5d73dfe8640ea445a161e8d9c1cac2fc491de65127619284804b3642647/globalmount: Input/output error (os error 5)
at control-plane/csi-driver/src/bin/node/filesystem_vol.rs:166
The volume shows:
core@sc-os-165-node3:~$ kubectl mayastor get volume 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6 -o yaml
spec:
num_replicas: 2
size: 32212254720
status: Created
target:
node: sc-os-165-node3.glcpdev.cloud.hpe.com
protocol: nvmf
frontend_nodes:
- name: sc-os-165-node3.glcpdev.cloud.hpe.com
nqn: nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com
uuid: 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6
topology:
pool_topology: !labelled
exclusion: {}
inclusion:
openebs.io/created-by: operator-diskpool
affinitykey: []
policy:
self_heal: true
thin: true
num_snapshots: 0
state:
target:
children:
- state: Online
uri: nvmf://10.245.250.167:8420/nqn.2019-05.io.openebs:2855148a-6812-4eed-ae9b-21e2d354bbfd?uuid=2855148a-6812-4eed-ae9b-21e2d354bbfd
- state: Online
uri: bdev:///9345ac3a-2c73-47e8-8229-983c0833f8aa?uuid=9345ac3a-2c73-47e8-8229-983c0833f8aa
deviceUri: nvmf://10.245.250.168:8420/nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6?hostnqn=nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com
node: sc-os-165-node3.glcpdev.cloud.hpe.com
rebuilds: 0
protocol: nvmf
size: 32212254720
state: Online
uuid: babd5cf3-d215-46b2-9667-51684b847c22
size: 32212254720
status: Online
uuid: 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6
replica_topology:
9345ac3a-2c73-47e8-8229-983c0833f8aa:
node: sc-os-165-node3.glcpdev.cloud.hpe.com
pool: sc-os-165-node3.glcpdev.cloud.hpe.com-disk
state: Online
child-status: Online
usage:
capacity: 32212254720
allocated: 947912704
allocated_snapshots: 0
allocated_all_snapshots: 0
2855148a-6812-4eed-ae9b-21e2d354bbfd:
node: sc-os-165-node2.glcpdev.cloud.hpe.com
pool: sc-os-165-node2.glcpdev.cloud.hpe.com-disk
state: Online
child-status: Online
usage:
capacity: 32212254720
allocated: 947912704
allocated_snapshots: 0
allocated_all_snapshots: 0
usage:
capacity: 32212254720
allocated: 947912704
allocated_replica: 947912704
allocated_snapshots: 0
allocated_all_snapshots: 0
total_allocated: 1895825408
total_allocated_replicas: 1895825408
total_allocated_snapshots: 0
Using the deviceUri to locate the device on the node:
core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 38
lrwxrwxrwx 1 root root 13 Aug 12 23:10 nvme-uuid.1c30f7e9-0182-46e7-9f87-f8758389c77b -> ../../nvme2n1
lrwxrwxrwx 1 root root 13 Aug 12 23:21 nvme-uuid.2dc6bf13-18ca-41ec-9b19-2e38bfbd4a89 -> ../../nvme3n1
lrwxrwxrwx 1 root root 14 Aug 12 23:21 nvme-uuid.3b38f55d-20f0-457c-8fde-7febe91174ee -> ../../nvme10n1
lrwxrwxrwx 1 root root 13 Aug 12 23:10 nvme-uuid.838b6fb1-c91f-4689-8a01-fbea59aef50c -> ../../nvme8n1
lrwxrwxrwx 1 root root 13 Aug 12 23:20 nvme-uuid.8efa4538-2a2f-43fc-9670-a44893879756 -> ../../nvme4n1
lrwxrwxrwx 1 root root 14 Aug 12 23:10 nvme-uuid.be874cf3-ef5a-4b2c-a99d-f3838be9cf45 -> ../../nvme12n1
Use the nvme-cli to manually attach, but it disappeared after a few min:
core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 3e
lrwxrwxrwx 1 root root 14 Aug 12 23:20 nvme-uuid.00263e9e-6837-41f0-a77c-c6227394bb47 -> ../../nvme17n1
lrwxrwxrwx 1 root root 14 Aug 13 13:20 nvme-uuid.3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6 -> ../../nvme38n1
lrwxrwxrwx 1 root root 13 Aug 12 23:10 nvme-uuid.4648d3bf-9c75-4969-a87d-6a959f95f3ec -> ../../nvme5n1
lrwxrwxrwx 1 root root 14 Aug 12 23:20 nvme-uuid.df645945-223e-4477-a208-73fe716d47b1 -> ../../nvme31n1
core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 3e
lrwxrwxrwx 1 root root 14 Aug 12 23:20 nvme-uuid.00263e9e-6837-41f0-a77c-c6227394bb47 -> ../../nvme17n1
lrwxrwxrwx 1 root root 13 Aug 12 23:10 nvme-uuid.4648d3bf-9c75-4969-a87d-6a959f95f3ec -> ../../nvme5n1
lrwxrwxrwx 1 root root 14 Aug 12 23:20 nvme-uuid.df645945-223e-4477-a208-73fe716d47b1 -> ../../nvme31n1
hmm I think the loop of connected/disconnected messages ie because staging is failing, and keeps being retried:
[2m2024-08-13T17:33:25.875732Z[0m [31mERROR[0m [1;31mcsi_node::filesystem_vol[0m[31m: [31mFailed to stage volume 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6: failed to mount device /dev/nvme38n1 onto /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/c7aaf5d73dfe8640ea445a161e8d9c1cac2fc491de65127619284804b3642647/globalmount: Input/output error (os error 5)[0m
[2;3mat[0m control-plane/csi-driver/src/bin/node/filesystem_vol.rs:166
But not sure why we are getting this error, could you share logs from data-plane on sc-os-165-node3.glcpdev.cloud.hpe.com
and also dmesg -T logs?
Attaching dmesg from node3.
Tried a dump system, got errors such as:
Could not fetch rebuild history for f08a1fa5-c8fa-4fd6-9870-3ed728652dd0, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))
Could not fetch rebuild history for faf1d26c-0833-4fa5-8142-1fca9f556fa9, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))
Also:
Completed collection of topology information
Error occurred while collecting logs
I tried moving the pod that uses 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6
to another node, it is still in the connect/disconnect loop.
Attaching it here: io-engine_node2.log.gz
EXT4-fs (nvme39n1): error loading journal
Ah, I think that's why it's happening! You need to go through the recovery procedure I wrote down on https://github.com/openebs/mayastor/issues/1713#issuecomment-2277466736
Tried a dump system, got errors such as:
Could not fetch rebuild history for f08a1fa5-c8fa-4fd6-9870-3ed728652dd0, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } }))) Could not fetch rebuild history for faf1d26c-0833-4fa5-8142-1fca9f556fa9, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))
It's a harmless error, we need to change dump tool to ignore this.
Also:
Completed collection of topology information Error occurred while collecting logs
This is not expected! Are you using the plugin from 2.5.1 or 2.7.0?
Could you be so kind to run it with RUST_LOG=debug?
RUST_LOG=debug kubectl-mayastor dump system
EXT4-fs (nvme39n1): error loading journal
Ah, I think that's why it's happening! You need to go through the recovery procedure I wrote down on #1713 (comment)
Does it mean the filesystem is corrupted? I tried to recover the corrupted FS, it's not going to work very well if the FS is used for persistent storage.
What/how can I prevent this from happening?
Also, in this setup, the volume is not staying connected even with the nvme-cli:
core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 3e
lrwxrwxrwx 1 root root 14 Aug 12 23:20 nvme-uuid.00263e9e-6837-41f0-a77c-c6227394bb47 -> ../../nvme17n1
lrwxrwxrwx 1 root root 14 Aug 13 13:20 nvme-uuid.3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6 -> ../../nvme38n1
lrwxrwxrwx 1 root root 13 Aug 12 23:10 nvme-uuid.4648d3bf-9c75-4969-a87d-6a959f95f3ec -> ../../nvme5n1
lrwxrwxrwx 1 root root 14 Aug 12 23:20 nvme-uuid.df645945-223e-4477-a208-73fe716d47b1 -> ../../nvme31n1
core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 3e
lrwxrwxrwx 1 root root 14 Aug 12 23:20 nvme-uuid.00263e9e-6837-41f0-a77c-c6227394bb47 -> ../../nvme17n1
lrwxrwxrwx 1 root root 13 Aug 12 23:10 nvme-uuid.4648d3bf-9c75-4969-a87d-6a959f95f3ec -> ../../nvme5n1
lrwxrwxrwx 1 root root 14 Aug 12 23:20 nvme-uuid.df645945-223e-4477-a208-73fe716d47b1 -> ../../nvme31n1
Also:
Completed collection of topology information Error occurred while collecting logs
This is not expected! Are you using the plugin from 2.5.1 or 2.7.0? Could you be so kind to run it with RUST_LOG=debug?
RUST_LOG=debug kubectl-mayastor dump system
I'm using 2.7.0 plugin. I'll try with DEBUG level and report back
Does it mean the filesystem is corrupted? I tried to recover the corrupted FS, it's not going to work very well if the FS is used for persistent storage. What/how can I prevent this from happening?
When the node crashes, there's a chance for filesystem corruption. You can run some fs utilities to try and fix it. There's also a way to mount without using the log IIRC, though in this case some data will surely be lost..
Also v2.5 also had a partial rebuild bug which we can't rule out as also having done some damage, if the volume have had partial rebuilds..
This is not expected! Are you using the plugin from 2.5.1 or 2.7.0? Could you be so kind to run it with RUST_LOG=debug?
RUST_LOG=debug kubectl-mayastor dump system
I'm using 2.7.0 plugin. I'll try with DEBUG level and report back
Great, thank you
The first dump "mayastor-2024-08-19--19-40-59-UTC.tar.gz" has the logs. Running again after 30 mins, didn't include the logs.
core@sc-os-80-node2:~$ RUST_LOG=debug kubectl-mayastor dump system
2024-08-19T19:40:58.934353Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services?&labelSelector=app%3Dapi-rest, otel.name: "list", otel.kind: "client"
2024-08-19T19:40:58.951060Z INFO kube_forward::http_forward: generated kube-api, uri: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy
at dependencies/control-plane/k8s/forward/src/http_forward.rs:54
2024-08-19T19:40:59.044983Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services?&labelSelector=app%3Dapi-rest, otel.name: "list", otel.kind: "client"
2024-08-19T19:40:59.069798Z INFO kube_forward::http_forward: generated kube-api, uri: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy
at dependencies/control-plane/k8s/forward/src/http_forward.rs:54
2024-08-19T19:40:59.206624Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services?&labelSelector=app%3Dloki, otel.name: "list", otel.kind: "client"
Failed to create loki client (Service 'app=loki' not found). Continuing...
2024-08-19T19:40:59.306927Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/kube-system, otel.name: "get", otel.kind: "client"
Collecting topology information...
2024-08-19T19:40:59.368357Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client"
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client"
2024-08-19T19:40:59.390353Z ERROR kube_client::client::builder: failed with status 503 Service Unavailable
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:151
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client"
2024-08-19T19:40:59.390391Z DEBUG openapi::clients::tower::configuration: failed to issue request, status: 503 Service Unavailable
at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:238
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
2024-08-19T19:40:59.395636Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client"
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.kind: "client"
2024-08-19T19:40:59.407334Z ERROR kube_client::client::builder: failed with status 503 Service Unavailable
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:151
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.kind: "client"
2024-08-19T19:40:59.407412Z DEBUG openapi::clients::tower::configuration: failed to issue request, status: 503 Service Unavailable
at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:238
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
2024-08-19T19:40:59.407736Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: "HTTP", otel.kind: "client"
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.kind: "client"
2024-08-19T19:40:59.418671Z ERROR kube_client::client::builder: failed with status 503 Service Unavailable
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:151
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: "HTTP", otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.kind: "client"
2024-08-19T19:40:59.418739Z DEBUG openapi::clients::tower::configuration: failed to issue request, status: 503 Service Unavailable
at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:238
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
2024-08-19T19:40:59.419007Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: "HTTP", otel.kind: "client"
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.kind: "client"
2024-08-19T19:40:59.430607Z ERROR kube_client::client::builder: failed with status 503 Service Unavailable
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:151
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: "HTTP", otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.kind: "client"
2024-08-19T19:40:59.430674Z DEBUG openapi::clients::tower::configuration: failed to issue request, status: 503 Service Unavailable
at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:238
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
Completed collection of topology information
2024-08-19T19:40:59.430961Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/pods?&fieldSelector=&labelSelector=openebs.io%2Flogging%3Dtrue&limit=100, otel.name: "list", otel.kind: "client"
2024-08-19T19:40:59.479820Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/nodes?, otel.name: "list", otel.kind: "client"
Error occurred while collecting logs
Collecting Kubernetes resources specific to mayastor service
Collecting Kubernetes events
2024-08-19T19:40:59.495643Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/events?&fieldSelector=&labelSelector=&limit=100, otel.name: "list", otel.kind: "client"
Collecting daemonsets configuration
2024-08-19T19:40:59.517809Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1/namespaces/mayastor/daemonsets?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"
Collecting deployments configuration
2024-08-19T19:40:59.524244Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1/namespaces/mayastor/deployments?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"
Collecting statefulsets configuration
2024-08-19T19:40:59.532673Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1/namespaces/mayastor/statefulsets?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"
Collecting Kubernetes disk pool resources
2024-08-19T19:40:59.539304Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/openebs.io/v1beta2/namespaces/mayastor/diskpools?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"
Collecting Kubernetes VolumeSnapshotClass resources
2024-08-19T19:40:59.544368Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.547118Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apiregistration.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.547225Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apiregistration.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.550384Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apps"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.550479Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.553055Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "events.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.553167Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/events.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.556219Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "authentication.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.556287Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/authentication.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.559261Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "authorization.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.559335Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/authorization.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.561333Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "autoscaling"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.561452Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/autoscaling/v2, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.564363Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/autoscaling/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.567057Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "batch"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.567146Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/batch/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.569538Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "certificates.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.569706Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/certificates.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.572543Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "networking.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.572652Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/networking.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.575690Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "policy"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.575766Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/policy/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.578563Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "rbac.authorization.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.578668Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/rbac.authorization.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.581528Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "storage.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.581717Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/storage.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.584742Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "admissionregistration.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.584855Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/admissionregistration.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.587688Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apiextensions.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.587792Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apiextensions.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.590593Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "scheduling.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.590711Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/scheduling.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.593489Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "coordination.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.593603Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/coordination.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.595867Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "node.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.595965Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/node.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.598715Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "discovery.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.598818Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/discovery.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.601934Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "flowcontrol.apiserver.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.602083Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/flowcontrol.apiserver.k8s.io/v1beta3, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.605336Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/flowcontrol.apiserver.k8s.io/v1beta2, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.612872Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "acme.cert-manager.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.613038Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/acme.cert-manager.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.615796Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "cert-manager.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.615925Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/cert-manager.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.618783Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "crd.projectcalico.org"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.618944Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/crd.projectcalico.org/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.622555Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "jaegertracing.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.622773Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/jaegertracing.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.625683Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "snapshot.storage.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.625850Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/snapshot.storage.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.628876Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "metallb.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.629039Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1beta2, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.631209Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1beta1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.634388Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1alpha1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.636841Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "openebs.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.636967Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/openebs.io/v1beta2, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.639283Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "metrics.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.639398Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metrics.k8s.io/v1beta1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.645410Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.648453Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.653107Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/snapshot.storage.k8s.io/v1/volumesnapshotclasses?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"
Collecting Kubernetes VolumeSnapshotContents resources
2024-08-19T19:40:59.670293Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.672139Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apiregistration.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.672253Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apiregistration.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.675039Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apps"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.675166Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.676722Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "events.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.676869Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/events.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.679762Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "authentication.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.679884Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/authentication.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.682442Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "authorization.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.682531Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/authorization.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.684919Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "autoscaling"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.685079Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/autoscaling/v2, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.688105Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/autoscaling/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.690560Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "batch"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.690662Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/batch/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.692375Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "certificates.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.692484Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/certificates.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.694577Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "networking.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.694672Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/networking.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.697023Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "policy"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.697137Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/policy/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.699327Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "rbac.authorization.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.699455Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/rbac.authorization.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.701534Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "storage.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.701699Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/storage.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.704467Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "admissionregistration.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.704591Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/admissionregistration.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.707106Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apiextensions.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.707198Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apiextensions.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.709726Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "scheduling.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.709834Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/scheduling.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.712617Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "coordination.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.712689Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/coordination.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.715114Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "node.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.715192Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/node.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.717943Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "discovery.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.718013Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/discovery.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.719552Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "flowcontrol.apiserver.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.719677Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/flowcontrol.apiserver.k8s.io/v1beta3, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.721411Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/flowcontrol.apiserver.k8s.io/v1beta2, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.723114Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "acme.cert-manager.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.723247Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/acme.cert-manager.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.724906Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "cert-manager.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.725030Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/cert-manager.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.726779Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "crd.projectcalico.org"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.726919Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/crd.projectcalico.org/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.728946Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "jaegertracing.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.729068Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/jaegertracing.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.731945Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "snapshot.storage.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.732071Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/snapshot.storage.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.733311Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "metallb.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.733427Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1beta2, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.734782Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1beta1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.736246Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1alpha1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.739056Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "openebs.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.739194Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/openebs.io/v1beta2, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.740830Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "metrics.k8s.io"
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80
2024-08-19T19:40:59.740939Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metrics.k8s.io/v1beta1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.745818Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.747540Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1, otel.name: "HTTP", otel.kind: "client"
2024-08-19T19:40:59.751085Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/snapshot.storage.k8s.io/v1/volumesnapshotcontents?&fieldSelector=&labelSelector=&limit=2, otel.name: "list", otel.kind: "client"
Collecting Kuberbetes pod resources
2024-08-19T19:40:59.767874Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/pods?&fieldSelector=&labelSelector=&limit=100, otel.name: "list", otel.kind: "client"
Completed collection of Kubernetes resource specific information
Collecting mayastor specific information from Etcd...
2024-08-19T19:40:59.814977Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/pods?&fieldSelector=status.phase%3DRunning&labelSelector=app%3Detcd, otel.name: "list", otel.kind: "client"
2024-08-19T19:40:59.847041Z ERROR kube_forward::port_forward: failed to forward connection, error: No pod 'app=etcd' available
at dependencies/control-plane/k8s/forward/src/port_forward.rs:81
Failed to collect etcd dump information, error: Etcd(GetPrefix { prefix: "/openebs.io/mayastor/apis/v0/clusters/b3ab0f6d-300d-47f6-837f-30ffd08418aa/namespaces/mayastor", source: GRpcStatus(Status { code: Unknown, message: "transport error", source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Kind(ConnectionReset)))) }) })
Completed collection of dump !!
core@sc-os-80-node2:~$ ls
mayastor-2024-08-19--19-12-06-UTC.tar.gz mayastor-2024-08-19--19-40-59-UTC.tar.gz snap verisign-enabled
core@sc-os-80-node2:~$ ls -al
total 624
drwxr-x--- 7 core core 4096 Aug 19 12:40 .
drwxr-xr-x 4 root root 4096 Jul 31 10:04 ..
drwx------ 3 core core 4096 Jul 31 10:37 .ansible
-rw------- 1 core core 96 Aug 6 14:16 .bash_history
-rw-r--r-- 1 core core 220 Jan 6 2022 .bash_logout
-rw-r--r-- 1 core core 3771 Jan 6 2022 .bashrc
drwx------ 2 core core 4096 Jul 31 10:37 .cache
drwxrwxrwx 3 root root 4096 Aug 19 12:10 .kube
-rw-rw-r-- 1 core core 552761 Aug 19 12:14 mayastor-2024-08-19--19-12-06-UTC.tar.gz
-rw-rw-r-- 1 core core 38594 Aug 19 12:40 mayastor-2024-08-19--19-40-59-UTC.tar.gz
-rw-r--r-- 1 core core 807 Jan 6 2022 .profile
drwxrwxrwx 2 root root 4096 Jul 31 11:15 snap
drwx------ 2 core core 4096 Jul 31 10:22 .ssh
-rw-r--r-- 1 core core 0 Jul 31 10:20 .sudo_as_admin_successful
-rw-r--r-- 1 root root 0 Jul 31 10:20 verisign-enabled
core@sc-os-80-node2:~$ tar -tvf mayastor-2024-08-19--19-40-59-UTC.tar.gz
drwxrwxr-x 1000/1000 0 2024-08-19 12:40 ./
-rw-rw-r-- 1000/1000 0 2024-08-19 12:40 etcd_dump
drwxrwxr-x 1000/1000 0 2024-08-19 12:40 k8s_resources
-rw-rw-r-- 1000/1000 264705 2024-08-19 12:40 k8s_resources/pods.yaml
-rw-rw-r-- 1000/1000 118686 2024-08-19 12:40 k8s_resources/k8s_events.json
-rw-rw-r-- 1000/1000 3 2024-08-19 12:40 k8s_resources/volume_snapshot_contents.yaml
drwxrwxr-x 1000/1000 0 2024-08-19 12:40 k8s_resources/configurations
-rw-rw-r-- 1000/1000 12560 2024-08-19 12:40 k8s_resources/configurations/mayastor-io-engine.yaml
-rw-rw-r-- 1000/1000 8531 2024-08-19 12:40 k8s_resources/configurations/mayastor-agent-ha-node.yaml
-rw-rw-r-- 1000/1000 11049 2024-08-19 12:40 k8s_resources/configurations/mayastor-csi-node.yaml
-rw-rw-r-- 1000/1000 9911 2024-08-19 12:40 k8s_resources/configurations/mayastor-agent-core.yaml
-rw-rw-r-- 1000/1000 7757 2024-08-19 12:40 k8s_resources/configurations/mayastor-localpv-provisioner.yaml
-rw-rw-r-- 1000/1000 12459 2024-08-19 12:40 k8s_resources/configurations/mayastor-csi-controller.yaml
-rw-rw-r-- 1000/1000 14936 2024-08-19 12:40 k8s_resources/configurations/mayastor-etcd.yaml
-rw-rw-r-- 1000/1000 6840 2024-08-19 12:40 k8s_resources/configurations/mayastor-operator-diskpool.yaml
-rw-rw-r-- 1000/1000 12104 2024-08-19 12:40 k8s_resources/configurations/mayastor-nats.yaml
-rw-rw-r-- 1000/1000 7982 2024-08-19 12:40 k8s_resources/configurations/mayastor-api-rest.yaml
-rw-rw-r-- 1000/1000 1704 2024-08-19 12:40 k8s_resources/k8s_disk_pools.yaml
-rw-rw-r-- 1000/1000 3 2024-08-19 12:40 k8s_resources/volume_snapshot_classes.yaml
-rw-rw-r-- 1000/1000 1099 2024-08-19 12:40 support_tool_logs.log
Update on the mayastor plug-in returning an error:
core@glop-nm-126-mem3:~$ RUST_LOG=debug kubectl-mayastor get volumes
2024-08-20T22:47:49.773676Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.244.130:16443/api/v1/namespaces/mayastor/services?&labelSelector=app%3Dapi-rest, otel.name: "list", otel.kind: "client"
2024-08-20T22:47:49.792212Z INFO kube_forward::http_forward: generated kube-api, uri: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy
at dependencies/control-plane/k8s/forward/src/http_forward.rs:54
2024-08-20T22:47:49.847900Z DEBUG kube_client::client::builder: requesting
at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.244.130:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client"
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client"
2024-08-20T22:47:59.848699Z DEBUG openapi::clients::tower::configuration: failed to issue request, error: request timed out
at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:241
in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client", otel.status_code: "ERROR"
Failed to list volumes. Error error in request: request timed out
List of running pods:
core@glop-nm-126-mem3:~$ kubectl get pods -n mayastor
NAME READY STATUS RESTARTS AGE
mayastor-agent-core-88bc8d8b9-8vk97 2/2 Running 0 46m
mayastor-agent-ha-node-vr9mb 1/1 Running 5 (20h ago) 20h
mayastor-agent-ha-node-z5zgh 1/1 Running 0 121m
mayastor-api-rest-d666fb647-6z7w7 1/1 Running 0 5h52m
mayastor-api-rest-d666fb647-9dv5m 1/1 Running 0 19h
mayastor-api-rest-d666fb647-fm97p 0/1 Pending 0 46m
mayastor-csi-controller-799f88d74d-hqtct 6/6 Running 0 5h46m
mayastor-csi-node-dn4sk 2/2 Running 0 121m
mayastor-csi-node-txhvn 2/2 Running 0 151m
mayastor-etcd-0 1/1 Running 0 5h46m
mayastor-etcd-1 1/1 Running 0 19h
mayastor-etcd-2 0/1 Pending 0 40m
mayastor-io-engine-2k44l 2/2 Running 0 121m
mayastor-io-engine-s5l92 2/2 Running 0 156m
mayastor-localpv-provisioner-796cdf4d5b-gpcn5 1/1 Running 0 41m
mayastor-nats-0 3/3 Running 0 5h46m
mayastor-nats-1 3/3 Running 0 40m
mayastor-nats-2 3/3 Running 0 40m
mayastor-operator-diskpool-5c894c69bf-nwjhj 1/1 Running 0 41m
Attaching agent-core logs: mayastor-agent-core-88bc8d8b9-8vk97.log.gz
mayastor-api-rest-d666fb647-9dv5m.log.gz mayastor-api-rest-d666fb647-6z7w7.log.gz
You seem to have configured 3 replicas for the api-rest, any reason for this?
Seems that one of them is in Pending
:
mayastor-api-rest-d666fb647-6z7w7 1/1 Running 0 5h52m
mayastor-api-rest-d666fb647-9dv5m 1/1 Running 0 19h
mayastor-api-rest-d666fb647-fm97p 0/1 Pending 0 46m
Also etcd has a pending pod:
mayastor-etcd-0 1/1 Running 0 5h46m
mayastor-etcd-1 1/1 Running 0 19h
mayastor-etcd-2 0/1 Pending 0 40m
Anyway, moving on to the error, is due to timeout:
[2m2024-08-20T22:48:17.698942Z[0m [31mERROR[0m [1;31mgrpc::operations::volume::client[0m[31m: [1;31merror[0m[31m: Aborted: status: Cancelled, message: "Timeout expired", details: [], metadata: MetadataMap { headers: {} }[0m
[2;3mat[0m control-plane/grpc/src/operations/volume/client.rs:76
[2m2024-08-20T22:48:46.785375Z[0m [32m INFO[0m [1;32mactix_web::middleware::logger[0m[32m: [32m172.23.101.192 "GET /v0/volumes?max_entries=200&starting_token=0 HTTP/1.1" 503 143 "-" "-" 54.687851[0m
[2;3mat[0m /build/cargo-vendor-dir/actix-web-4.4.0/src/middleware/logger.rs:420
But on the agents-core there's nothing there which would explain the timeout... I can see requests taking minutes to complete though:
[2m2024-08-20T22:51:27.585545Z[0m [32m INFO[0m [1;32mactix_web::middleware::logger[0m[32m: [32m172.23.138.198 "GET /v0/nodes/glop-nm-126-mem3.glcpdev.cloud.hpe.com/pools/glop-nm-126-mem3.glcpdev.cloud.hpe.com-disk HTTP/1.1" 503 143 "-" "-" 144.404435[0m
[2;3mat[0m /build/cargo-vendor-dir/actix-web-4.4.0/src/middleware/logger.rs:420
[2m2024-08-20T22:51:27.585601Z[0m [32m INFO[0m [1;32mactix_web::middleware::logger[0m[32m: [32m172.23.138.198 "GET /v0/nodes/glop-nm-126-mem3.glcpdev.cloud.hpe.com/pools/glop-nm-126-mem3.glcpdev.cloud.hpe.com-disk HTTP/1.1" 503 143 "-" "-" 49.604255[0m
[2;3mat[0m /build/cargo-vendor-dir/actix-web-4.4.0/src/middleware/logger.rs:420
@Abhinandan-Purkait any ideas here? Maybe we should enable tracing?
@Abhinandan-Purkait any ideas here?
I think it's because the rest-api isn't supposed to run with multiple instances, as it doesn't have any sort of loadbalancing and or leader/slave mechanism. Timeouts probably because the same request is being served by different instances of rest-api and the locks held at core-agent puts one on wait?
I believe it has nothing to do with the connectivity or computing time due to load?
@veenadong could you please change agents.core.logLevel to debug? This should help provide more information. If it doesn't we can try enabling jaeger to get more detailed information
@Abhinandan-Purkait any ideas here?
I think it's because the rest-api isn't supposed to run with multiple instances, as it doesn't have any sort of loadbalancing and or leader/slave mechanism. Timeouts probably because the same request is being served by different instances of rest-api and the locks held at core-agent puts one on wait?
I believe it has nothing to do with the connectivity or computing time due to load?
I just checked and I think the above could not be, as in a Kubernetes ClusterIP service setup, it is highly unlikely for two different pods to receive the exact same request under normal circumstances, as it kind of does load balancing by default, round robin.
Yeah, it would be good to see debug logs.
Upgraded from 2.5.1 to 2.7.0. After the upgrade, 1 node in a 3 node cluster crashed.
Currently running nodes:
List of pods running in mayastor namespace:
Versions of all the pods:
The mayastor plugin didn't capture any of the logs, so I captured using kubectl:
mayastor.log.tar.gz
mayastor-2024-08-13--16-56-32-UTC.tar.gz