search

openebs / mayastor

Dynamically provision Stateful Persistent Replicated Cluster-wide Fabric Volumes & Filesystems for Kubernetes that is provisioned from an optimized NVME SPDK backend data storage stack.
Apache License 2.0
742 stars 106 forks source link

Diskpools are in "Created" state after 1 node crashed #1719

Open veenadong opened 2 months ago

veenadong commented 2 months ago

Upgraded from 2.5.1 to 2.7.0. After the upgrade, 1 node in a 3 node cluster crashed.

  1. Mayastor plugin is getting a rest timeout:

core@sc-os-165-node2:~$ kubectl-mayastor get nodes Failed to list nodes. Error error in request: request timed out

core@sc-os-165-node3:~$ kubectl mayastor get pools Failed to list pools. Error error in request: request timed out

  1. All diskpools are in "Created" state without any other status:

core@sc-os-165-node3:~$ kubectl get dsp -A NAMESPACE NAME NODE STATE POOL_STATUS CAPACITY USED AVAILABLE mayastor sc-os-165-node1.glcpdev.cloud.hpe.com-disk sc-os-165-node1.glcpdev.cloud.hpe.com Created 0 0 0 mayastor sc-os-165-node2.glcpdev.cloud.hpe.com-disk sc-os-165-node2.glcpdev.cloud.hpe.com Created 0 0 0 mayastor sc-os-165-node3.glcpdev.cloud.hpe.com-disk sc-os-165-node3.glcpdev.cloud.hpe.com Created 0 0 0

Currently running nodes:

core@sc-os-165-node3:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION sc-os-165-node1.glcpdev.cloud.hpe.com NotReady 27d v1.28.7 sc-os-165-node2.glcpdev.cloud.hpe.com Ready 26d v1.28.7 sc-os-165-node3.glcpdev.cloud.hpe.com Ready 26d v1.28.7

List of pods running in mayastor namespace:

mayastor-agent-core-88bc8d8b9-v9k9n 2/2 Running 7 (12h ago) 16h 172.23.155.217 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-agent-ha-node-857bb 1/1 Running 0 12h 10.245.250.168 sc-os-165-node3.glcpdev.cloud.hpe.com mayastor-agent-ha-node-c5bqw 1/1 Running 1 (12h ago) 16h 10.245.250.167 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-api-rest-d666fb647-9xbfb 1/1 Running 1 (12h ago) 16h 172.23.155.163 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-csi-controller-799f88d74d-rqcm4 6/6 Running 6 (12h ago) 16h 10.245.250.167 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-csi-node-mmnw8 2/2 Running 0 12h 10.245.250.168 sc-os-165-node3.glcpdev.cloud.hpe.com mayastor-csi-node-pgkg6 2/2 Running 2 (12h ago) 16h 10.245.250.167 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-etcd-0 0/1 Pending 0 14h mayastor-etcd-1 1/1 Running 5 (12h ago) 16h 172.23.155.140 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-etcd-2 1/1 Running 0 12h 172.23.223.112 sc-os-165-node3.glcpdev.cloud.hpe.com mayastor-io-engine-7lmgp 2/2 Running 0 12h 10.245.250.168 sc-os-165-node3.glcpdev.cloud.hpe.com mayastor-io-engine-vfglv 2/2 Running 3 (12h ago) 16h 10.245.250.167 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-localpv-provisioner-796cdf4d5b-78w2h 1/1 Running 6 (12h ago) 16h 172.23.155.212 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-nats-0 3/3 Running 3 (12h ago) 14h 172.23.155.196 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-nats-1 3/3 Running 0 12h 172.23.156.32 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-nats-2 3/3 Running 3 (12h ago) 16h 172.23.155.146 sc-os-165-node2.glcpdev.cloud.hpe.com mayastor-operator-diskpool-5c894c69bf-mjpjq 1/1 Running 0 12h 172.23.156.8 sc-os-165-node2.glcpdev.cloud.hpe.com

Versions of all the pods:

mayastor-agent-core-88bc8d8b9-v9k9n:docker.io/openebs/mayastor-agent-core:v2.7.0, docker.io/openebs/mayastor-agent-ha-cluster:v2.7.0, mayastor-agent-ha-node-857bb:docker.io/openebs/mayastor-agent-ha-node:v2.7.0, mayastor-agent-ha-node-c5bqw:docker.io/openebs/mayastor-agent-ha-node:v2.7.0, mayastor-api-rest-d666fb647-9xbfb:docker.io/openebs/mayastor-api-rest:v2.7.0, mayastor-csi-controller-799f88d74d-rqcm4:registry.k8s.io/sig-storage/csi-provisioner:v3.5.0, registry.k8s.io/sig-storage/csi-attacher:v4.3.0, registry.k8s.io/sig-storage/csi-snapshotter:v6.3.3, registry.k8s.io/sig-storage/snapshot-controller:v6.3.3, registry.k8s.io/sig-storage/csi-resizer:v1.9.3, docker.io/openebs/mayastor-csi-controller:v2.7.0, mayastor-csi-node-mmnw8:docker.io/openebs/mayastor-csi-node:v2.7.0, registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0, mayastor-csi-node-pgkg6:docker.io/openebs/mayastor-csi-node:v2.7.0, registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0, mayastor-etcd-0:docker.io/bitnami/etcd:3.5.6-debian-11-r10, mayastor-etcd-1:docker.io/bitnami/etcd:3.5.6-debian-11-r10, mayastor-etcd-2:docker.io/bitnami/etcd:3.5.6-debian-11-r10, mayastor-io-engine-7lmgp:docker.io/openebs/mayastor-metrics-exporter-io-engine:v2.7.0, docker.io/openebs/mayastor-io-engine:v2.7.0, mayastor-io-engine-vfglv:docker.io/openebs/mayastor-metrics-exporter-io-engine:v2.7.0, docker.io/openebs/mayastor-io-engine:v2.7.0, mayastor-localpv-provisioner-796cdf4d5b-78w2h:openebs/provisioner-localpv:4.1.0, mayastor-nats-0:nats:2.9.17-alpine, natsio/nats-server-config-reloader:0.10.1, natsio/prometheus-nats-exporter:0.11.0, mayastor-nats-1:nats:2.9.17-alpine, natsio/nats-server-config-reloader:0.10.1, natsio/prometheus-nats-exporter:0.11.0, mayastor-nats-2:nats:2.9.17-alpine, natsio/nats-server-config-reloader:0.10.1, natsio/prometheus-nats-exporter:0.11.0, mayastor-operator-diskpool-5c894c69bf-mjpjq:docker.io/openebs/mayastor-operator-diskpool:v2.7.0

The mayastor plugin didn't capture any of the logs, so I captured using kubectl:

mayastor.log.tar.gz

mayastor-2024-08-13--16-56-32-UTC.tar.gz

veenadong commented 2 months ago

Update: Recovered api-rest by restarting the pod. This got the diskpools back. However, io-engine continues to log:

[2024-08-13T18:57:26.262907584+00:00  INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:324] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' connected to subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T18:57:30.982486681+00:00  INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:344] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' disconnected from subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T18:59:33.849688174+00:00  INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:324] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' connected to subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T18:59:37.633400695+00:00  INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:344] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' disconnected from subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T19:00:04.647449691+00:00  INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:324] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' connected to subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T19:01:42.336838311+00:00  INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:344] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' disconnected from subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T19:03:45.060224862+00:00  INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:324] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' connected to subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'
[2024-08-13T19:03:48.878250758+00:00  INFO io_engine::subsys::nvmf::subsystem:subsystem.rs:344] Host 'nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com' disconnected from subsystem 'nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' on nexus 'Nexus '3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6' [open]'

The pod trying to use this volume:

Events:
  Type     Reason       Age                    From     Message
  ----     ------       ----                   ----     -------
  Warning  FailedMount  3m49s (x361 over 12h)  kubelet  MountVolume.MountDevice failed for volume "pvc-3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6" : rpc error: code = Internal desc = Failed to stage volume 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6: failed to mount device /dev/nvme38n1 onto /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/c7aaf5d73dfe8640ea445a161e8d9c1cac2fc491de65127619284804b3642647/globalmount: Input/output error (os error 5)

Another pod is logging:

Events:
  Type     Reason       Age                  From     Message
  ----     ------       ----                 ----     -------
  Warning  FailedMount  11s (x234 over 12h)  kubelet  MountVolume.MountDevice failed for volume "pvc-1b693021-bce1-4490-9916-2a44c3e02a05" : rpc error: code = Unavailable desc = Failed to stage volume 1b693021-bce1-4490-9916-2a44c3e02a05: device attach timeout

While the volume is showing as online:

core@sc-os-165-node3:~$ kubectl mayastor get volume 1b693021-bce1-4490-9916-2a44c3e02a05
 ID                                    REPLICAS  TARGET-NODE                            ACCESSIBILITY  STATUS  SIZE   THIN-PROVISIONED  ALLOCATED  SNAPSHOTS  SOURCE
 1b693021-bce1-4490-9916-2a44c3e02a05  2         sc-os-165-node3.glcpdev.cloud.hpe.com  nvmf           Online  10GiB  true              1.4GiB     0          <none>
core@sc-os-165-node3:~$ kubectl mayastor get volume 1b693021-bce1-4490-9916-2a44c3e02a05 -o yaml
spec:
  num_replicas: 2
  size: 10737418240
  status: Created
  target:
    node: sc-os-165-node3.glcpdev.cloud.hpe.com
    protocol: nvmf
    frontend_nodes:
    - name: sc-os-165-node3.glcpdev.cloud.hpe.com
      nqn: nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com
  uuid: 1b693021-bce1-4490-9916-2a44c3e02a05
  topology:
    pool_topology: !labelled
      exclusion: {}
      inclusion:
        openebs.io/created-by: operator-diskpool
      affinitykey: []
  policy:
    self_heal: true
  thin: true
  num_snapshots: 0
state:
  target:
    children:
    - state: Online
      uri: bdev:///d89ffaf8-868b-418d-9068-68fd5df377b1?uuid=d89ffaf8-868b-418d-9068-68fd5df377b1
    - state: Online
      uri: nvmf://10.245.250.167:8420/nqn.2019-05.io.openebs:29e303ed-91f3-4888-bd9f-cf11186fe3a7?uuid=29e303ed-91f3-4888-bd9f-cf11186fe3a7
    deviceUri: nvmf://10.245.250.168:8420/nqn.2019-05.io.openebs:1b693021-bce1-4490-9916-2a44c3e02a05?hostnqn=nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com
    node: sc-os-165-node3.glcpdev.cloud.hpe.com
    rebuilds: 0
    protocol: nvmf
    size: 10737418240
    state: Online
    uuid: 6bde92b3-a0e0-4194-a797-0201ec250168
  size: 10737418240
  status: Online
  uuid: 1b693021-bce1-4490-9916-2a44c3e02a05
  replica_topology:
    d89ffaf8-868b-418d-9068-68fd5df377b1:
      node: sc-os-165-node3.glcpdev.cloud.hpe.com
      pool: sc-os-165-node3.glcpdev.cloud.hpe.com-disk
      state: Online
      child-status: Online
      usage:
        capacity: 10737418240
        allocated: 1472200704
        allocated_snapshots: 0
        allocated_all_snapshots: 0
    29e303ed-91f3-4888-bd9f-cf11186fe3a7:
      node: sc-os-165-node2.glcpdev.cloud.hpe.com
      pool: sc-os-165-node2.glcpdev.cloud.hpe.com-disk
      state: Online
      child-status: Online
      usage:
        capacity: 10737418240
        allocated: 1472200704
        allocated_snapshots: 0
        allocated_all_snapshots: 0
  usage:
    capacity: 10737418240
    allocated: 1472200704
    allocated_replica: 1472200704
    allocated_snapshots: 0
    allocated_all_snapshots: 0
    total_allocated: 2944401408
    total_allocated_replicas: 2944401408
    total_allocated_snapshots: 0
veenadong commented 2 months ago

Noticed errors in csi-node:

 2024-08-13T19:54:21.383875Z ERROR csi_node::filesystem_vol: Failed to stage volume 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6: failed to mount device /dev/nvme38n1 onto /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/c7aaf5d73dfe8640ea445a161e8d9c1cac2fc491de65127619284804b3642647/globalmount: Input/output error (os error 5)
    at control-plane/csi-driver/src/bin/node/filesystem_vol.rs:166

The volume shows:

core@sc-os-165-node3:~$ kubectl mayastor get volume 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6 -o yaml
spec:
  num_replicas: 2
  size: 32212254720
  status: Created
  target:
    node: sc-os-165-node3.glcpdev.cloud.hpe.com
    protocol: nvmf
    frontend_nodes:
    - name: sc-os-165-node3.glcpdev.cloud.hpe.com
      nqn: nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com
  uuid: 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6
  topology:
    pool_topology: !labelled
      exclusion: {}
      inclusion:
        openebs.io/created-by: operator-diskpool
      affinitykey: []
  policy:
    self_heal: true
  thin: true
  num_snapshots: 0
state:
  target:
    children:
    - state: Online
      uri: nvmf://10.245.250.167:8420/nqn.2019-05.io.openebs:2855148a-6812-4eed-ae9b-21e2d354bbfd?uuid=2855148a-6812-4eed-ae9b-21e2d354bbfd
    - state: Online
      uri: bdev:///9345ac3a-2c73-47e8-8229-983c0833f8aa?uuid=9345ac3a-2c73-47e8-8229-983c0833f8aa
    deviceUri: nvmf://10.245.250.168:8420/nqn.2019-05.io.openebs:3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6?hostnqn=nqn.2019-05.io.openebs:node-name:sc-os-165-node3.glcpdev.cloud.hpe.com
    node: sc-os-165-node3.glcpdev.cloud.hpe.com
    rebuilds: 0
    protocol: nvmf
    size: 32212254720
    state: Online
    uuid: babd5cf3-d215-46b2-9667-51684b847c22
  size: 32212254720
  status: Online
  uuid: 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6
  replica_topology:
    9345ac3a-2c73-47e8-8229-983c0833f8aa:
      node: sc-os-165-node3.glcpdev.cloud.hpe.com
      pool: sc-os-165-node3.glcpdev.cloud.hpe.com-disk
      state: Online
      child-status: Online
      usage:
        capacity: 32212254720
        allocated: 947912704
        allocated_snapshots: 0
        allocated_all_snapshots: 0
    2855148a-6812-4eed-ae9b-21e2d354bbfd:
      node: sc-os-165-node2.glcpdev.cloud.hpe.com
      pool: sc-os-165-node2.glcpdev.cloud.hpe.com-disk
      state: Online
      child-status: Online
      usage:
        capacity: 32212254720
        allocated: 947912704
        allocated_snapshots: 0
        allocated_all_snapshots: 0
  usage:
    capacity: 32212254720
    allocated: 947912704
    allocated_replica: 947912704
    allocated_snapshots: 0
    allocated_all_snapshots: 0
    total_allocated: 1895825408
    total_allocated_replicas: 1895825408
    total_allocated_snapshots: 0

Using the deviceUri to locate the device on the node:

core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 38
lrwxrwxrwx 1 root root   13 Aug 12 23:10 nvme-uuid.1c30f7e9-0182-46e7-9f87-f8758389c77b -> ../../nvme2n1
lrwxrwxrwx 1 root root   13 Aug 12 23:21 nvme-uuid.2dc6bf13-18ca-41ec-9b19-2e38bfbd4a89 -> ../../nvme3n1
lrwxrwxrwx 1 root root   14 Aug 12 23:21 nvme-uuid.3b38f55d-20f0-457c-8fde-7febe91174ee -> ../../nvme10n1
lrwxrwxrwx 1 root root   13 Aug 12 23:10 nvme-uuid.838b6fb1-c91f-4689-8a01-fbea59aef50c -> ../../nvme8n1
lrwxrwxrwx 1 root root   13 Aug 12 23:20 nvme-uuid.8efa4538-2a2f-43fc-9670-a44893879756 -> ../../nvme4n1
lrwxrwxrwx 1 root root   14 Aug 12 23:10 nvme-uuid.be874cf3-ef5a-4b2c-a99d-f3838be9cf45 -> ../../nvme12n1

Use the nvme-cli to manually attach, but it disappeared after a few min:

core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 3e
lrwxrwxrwx 1 root root   14 Aug 12 23:20 nvme-uuid.00263e9e-6837-41f0-a77c-c6227394bb47 -> ../../nvme17n1
lrwxrwxrwx 1 root root   14 Aug 13 13:20 nvme-uuid.3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6 -> ../../nvme38n1
lrwxrwxrwx 1 root root   13 Aug 12 23:10 nvme-uuid.4648d3bf-9c75-4969-a87d-6a959f95f3ec -> ../../nvme5n1
lrwxrwxrwx 1 root root   14 Aug 12 23:20 nvme-uuid.df645945-223e-4477-a208-73fe716d47b1 -> ../../nvme31n1
core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 3e
lrwxrwxrwx 1 root root   14 Aug 12 23:20 nvme-uuid.00263e9e-6837-41f0-a77c-c6227394bb47 -> ../../nvme17n1
lrwxrwxrwx 1 root root   13 Aug 12 23:10 nvme-uuid.4648d3bf-9c75-4969-a87d-6a959f95f3ec -> ../../nvme5n1
lrwxrwxrwx 1 root root   14 Aug 12 23:20 nvme-uuid.df645945-223e-4477-a208-73fe716d47b1 -> ../../nvme31n1
tiagolobocastro commented 2 months ago

hmm I think the loop of connected/disconnected messages ie because staging is failing, and keeps being retried:

  2024-08-13T17:33:25.875732Z ERROR csi_node::filesystem_vol: Failed to stage volume 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6: failed to mount device /dev/nvme38n1 onto /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/c7aaf5d73dfe8640ea445a161e8d9c1cac2fc491de65127619284804b3642647/globalmount: Input/output error (os error 5)
    at control-plane/csi-driver/src/bin/node/filesystem_vol.rs:166

But not sure why we are getting this error, could you share logs from data-plane on sc-os-165-node3.glcpdev.cloud.hpe.com and also dmesg -T logs?

veenadong commented 2 months ago

Attaching dmesg from node3.

Tried a dump system, got errors such as:

Could not fetch rebuild history for f08a1fa5-c8fa-4fd6-9870-3ed728652dd0, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))
Could not fetch rebuild history for faf1d26c-0833-4fa5-8142-1fca9f556fa9, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))

Also:

Completed collection of topology information
Error occurred while collecting logs

dmesg_node3.log.gz

I tried moving the pod that uses 3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6 to another node, it is still in the connect/disconnect loop.

Attaching it here: io-engine_node2.log.gz

dmesg_node2.log.gz

tiagolobocastro commented 2 months ago

EXT4-fs (nvme39n1): error loading journal

Ah, I think that's why it's happening! You need to go through the recovery procedure I wrote down on https://github.com/openebs/mayastor/issues/1713#issuecomment-2277466736

Tried a dump system, got errors such as:

Could not fetch rebuild history for f08a1fa5-c8fa-4fd6-9870-3ed728652dd0, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))
Could not fetch rebuild history for faf1d26c-0833-4fa5-8142-1fca9f556fa9, error: RestJsonError(Response(Expected(ResponseContent { status: 412, body: RestJsonError { details: "", message: "volume target not available", kind: FailedPrecondition } })))

It's a harmless error, we need to change dump tool to ignore this.

Also:

Completed collection of topology information
Error occurred while collecting logs

This is not expected! Are you using the plugin from 2.5.1 or 2.7.0? Could you be so kind to run it with RUST_LOG=debug? RUST_LOG=debug kubectl-mayastor dump system

veenadong commented 2 months ago

EXT4-fs (nvme39n1): error loading journal

Ah, I think that's why it's happening! You need to go through the recovery procedure I wrote down on #1713 (comment)

Does it mean the filesystem is corrupted? I tried to recover the corrupted FS, it's not going to work very well if the FS is used for persistent storage.
What/how can I prevent this from happening?

Also, in this setup, the volume is not staying connected even with the nvme-cli:

core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 3e
lrwxrwxrwx 1 root root   14 Aug 12 23:20 nvme-uuid.00263e9e-6837-41f0-a77c-c6227394bb47 -> ../../nvme17n1
lrwxrwxrwx 1 root root   14 Aug 13 13:20 nvme-uuid.3e0de6f8-526b-4fe4-b2a4-35f2a471d2a6 -> ../../nvme38n1
lrwxrwxrwx 1 root root   13 Aug 12 23:10 nvme-uuid.4648d3bf-9c75-4969-a87d-6a959f95f3ec -> ../../nvme5n1
lrwxrwxrwx 1 root root   14 Aug 12 23:20 nvme-uuid.df645945-223e-4477-a208-73fe716d47b1 -> ../../nvme31n1
core@sc-os-165-node3:~$ sudo ls -al /dev/disk/by-id/ | grep 3e
lrwxrwxrwx 1 root root   14 Aug 12 23:20 nvme-uuid.00263e9e-6837-41f0-a77c-c6227394bb47 -> ../../nvme17n1
lrwxrwxrwx 1 root root   13 Aug 12 23:10 nvme-uuid.4648d3bf-9c75-4969-a87d-6a959f95f3ec -> ../../nvme5n1
lrwxrwxrwx 1 root root   14 Aug 12 23:20 nvme-uuid.df645945-223e-4477-a208-73fe716d47b1 -> ../../nvme31n1

Also:

Completed collection of topology information
Error occurred while collecting logs

This is not expected! Are you using the plugin from 2.5.1 or 2.7.0? Could you be so kind to run it with RUST_LOG=debug? RUST_LOG=debug kubectl-mayastor dump system

I'm using 2.7.0 plugin. I'll try with DEBUG level and report back

tiagolobocastro commented 2 months ago

Does it mean the filesystem is corrupted? I tried to recover the corrupted FS, it's not going to work very well if the FS is used for persistent storage. What/how can I prevent this from happening?

When the node crashes, there's a chance for filesystem corruption. You can run some fs utilities to try and fix it. There's also a way to mount without using the log IIRC, though in this case some data will surely be lost..

Also v2.5 also had a partial rebuild bug which we can't rule out as also having done some damage, if the volume have had partial rebuilds..

This is not expected! Are you using the plugin from 2.5.1 or 2.7.0? Could you be so kind to run it with RUST_LOG=debug? RUST_LOG=debug kubectl-mayastor dump system

I'm using 2.7.0 plugin. I'll try with DEBUG level and report back

Great, thank you

veenadong commented 2 months ago

The first dump "mayastor-2024-08-19--19-40-59-UTC.tar.gz" has the logs. Running again after 30 mins, didn't include the logs.

core@sc-os-80-node2:~$ RUST_LOG=debug kubectl-mayastor dump system
  2024-08-19T19:40:58.934353Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services?&labelSelector=app%3Dapi-rest, otel.name: "list", otel.kind: "client"

  2024-08-19T19:40:58.951060Z  INFO kube_forward::http_forward: generated kube-api, uri: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy
    at dependencies/control-plane/k8s/forward/src/http_forward.rs:54

  2024-08-19T19:40:59.044983Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services?&labelSelector=app%3Dapi-rest, otel.name: "list", otel.kind: "client"

  2024-08-19T19:40:59.069798Z  INFO kube_forward::http_forward: generated kube-api, uri: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy
    at dependencies/control-plane/k8s/forward/src/http_forward.rs:54

  2024-08-19T19:40:59.206624Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services?&labelSelector=app%3Dloki, otel.name: "list", otel.kind: "client"

Failed to create loki client (Service 'app=loki' not found). Continuing...
  2024-08-19T19:40:59.306927Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/kube-system, otel.name: "get", otel.kind: "client"

Collecting topology information...
  2024-08-19T19:40:59.368357Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client"
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client"

  2024-08-19T19:40:59.390353Z ERROR kube_client::client::builder: failed with status 503 Service Unavailable
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:151
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client"

  2024-08-19T19:40:59.390391Z DEBUG openapi::clients::tower::configuration: failed to issue request, status: 503 Service Unavailable
    at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:238
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503

  2024-08-19T19:40:59.395636Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client"
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.kind: "client"

  2024-08-19T19:40:59.407334Z ERROR kube_client::client::builder: failed with status 503 Service Unavailable
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:151
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.kind: "client"

  2024-08-19T19:40:59.407412Z DEBUG openapi::clients::tower::configuration: failed to issue request, status: 503 Service Unavailable
    at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:238
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes/snapshots?max_entries=200&starting_token=0, otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503

  2024-08-19T19:40:59.407736Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: "HTTP", otel.kind: "client"
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.kind: "client"

  2024-08-19T19:40:59.418671Z ERROR kube_client::client::builder: failed with status 503 Service Unavailable
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:151
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: "HTTP", otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.kind: "client"

  2024-08-19T19:40:59.418739Z DEBUG openapi::clients::tower::configuration: failed to issue request, status: 503 Service Unavailable
    at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:238
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/pools, otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503

  2024-08-19T19:40:59.419007Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: "HTTP", otel.kind: "client"
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.kind: "client"

  2024-08-19T19:40:59.430607Z ERROR kube_client::client::builder: failed with status 503 Service Unavailable
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:151
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: "HTTP", otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.kind: "client"

  2024-08-19T19:40:59.430674Z DEBUG openapi::clients::tower::configuration: failed to issue request, status: 503 Service Unavailable
    at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:238
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/nodes, otel.kind: "client", http.status_code: 503, otel.status_code: "ERROR", otel.status_code: "ERROR", http.status_code: 503

Completed collection of topology information
  2024-08-19T19:40:59.430961Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/pods?&fieldSelector=&labelSelector=openebs.io%2Flogging%3Dtrue&limit=100, otel.name: "list", otel.kind: "client"

  2024-08-19T19:40:59.479820Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/nodes?, otel.name: "list", otel.kind: "client"

Error occurred while collecting logs
Collecting Kubernetes resources specific to mayastor service
     Collecting Kubernetes events
  2024-08-19T19:40:59.495643Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/events?&fieldSelector=&labelSelector=&limit=100, otel.name: "list", otel.kind: "client"

     Collecting daemonsets configuration
  2024-08-19T19:40:59.517809Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1/namespaces/mayastor/daemonsets?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"

     Collecting deployments configuration
  2024-08-19T19:40:59.524244Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1/namespaces/mayastor/deployments?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"

     Collecting statefulsets configuration
  2024-08-19T19:40:59.532673Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1/namespaces/mayastor/statefulsets?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"

     Collecting Kubernetes disk pool resources
  2024-08-19T19:40:59.539304Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/openebs.io/v1beta2/namespaces/mayastor/diskpools?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"

     Collecting Kubernetes VolumeSnapshotClass resources
  2024-08-19T19:40:59.544368Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.547118Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apiregistration.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.547225Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apiregistration.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.550384Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apps"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.550479Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.553055Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "events.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.553167Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/events.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.556219Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "authentication.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.556287Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/authentication.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.559261Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "authorization.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.559335Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/authorization.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.561333Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "autoscaling"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.561452Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/autoscaling/v2, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.564363Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/autoscaling/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.567057Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "batch"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.567146Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/batch/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.569538Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "certificates.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.569706Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/certificates.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.572543Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "networking.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.572652Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/networking.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.575690Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "policy"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.575766Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/policy/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.578563Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "rbac.authorization.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.578668Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/rbac.authorization.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.581528Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "storage.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.581717Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/storage.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.584742Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "admissionregistration.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.584855Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/admissionregistration.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.587688Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apiextensions.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.587792Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apiextensions.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.590593Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "scheduling.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.590711Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/scheduling.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.593489Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "coordination.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.593603Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/coordination.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.595867Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "node.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.595965Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/node.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.598715Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "discovery.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.598818Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/discovery.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.601934Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "flowcontrol.apiserver.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.602083Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/flowcontrol.apiserver.k8s.io/v1beta3, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.605336Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/flowcontrol.apiserver.k8s.io/v1beta2, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.612872Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "acme.cert-manager.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.613038Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/acme.cert-manager.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.615796Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "cert-manager.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.615925Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/cert-manager.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.618783Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "crd.projectcalico.org"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.618944Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/crd.projectcalico.org/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.622555Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "jaegertracing.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.622773Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/jaegertracing.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.625683Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "snapshot.storage.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.625850Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/snapshot.storage.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.628876Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "metallb.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.629039Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1beta2, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.631209Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1beta1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.634388Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1alpha1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.636841Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "openebs.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.636967Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/openebs.io/v1beta2, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.639283Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "metrics.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.639398Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metrics.k8s.io/v1beta1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.645410Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.648453Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.653107Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/snapshot.storage.k8s.io/v1/volumesnapshotclasses?&fieldSelector=&labelSelector=, otel.name: "list", otel.kind: "client"

     Collecting Kubernetes VolumeSnapshotContents resources
  2024-08-19T19:40:59.670293Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.672139Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apiregistration.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.672253Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apiregistration.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.675039Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apps"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.675166Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apps/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.676722Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "events.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.676869Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/events.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.679762Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "authentication.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.679884Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/authentication.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.682442Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "authorization.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.682531Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/authorization.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.684919Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "autoscaling"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.685079Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/autoscaling/v2, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.688105Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/autoscaling/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.690560Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "batch"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.690662Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/batch/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.692375Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "certificates.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.692484Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/certificates.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.694577Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "networking.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.694672Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/networking.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.697023Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "policy"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.697137Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/policy/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.699327Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "rbac.authorization.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.699455Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/rbac.authorization.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.701534Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "storage.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.701699Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/storage.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.704467Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "admissionregistration.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.704591Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/admissionregistration.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.707106Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "apiextensions.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.707198Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/apiextensions.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.709726Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "scheduling.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.709834Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/scheduling.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.712617Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "coordination.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.712689Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/coordination.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.715114Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "node.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.715192Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/node.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.717943Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "discovery.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.718013Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/discovery.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.719552Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "flowcontrol.apiserver.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.719677Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/flowcontrol.apiserver.k8s.io/v1beta3, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.721411Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/flowcontrol.apiserver.k8s.io/v1beta2, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.723114Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "acme.cert-manager.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.723247Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/acme.cert-manager.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.724906Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "cert-manager.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.725030Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/cert-manager.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.726779Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "crd.projectcalico.org"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.726919Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/crd.projectcalico.org/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.728946Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "jaegertracing.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.729068Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/jaegertracing.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.731945Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "snapshot.storage.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.732071Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/snapshot.storage.k8s.io/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.733311Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "metallb.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.733427Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1beta2, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.734782Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1beta1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.736246Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metallb.io/v1alpha1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.739056Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "openebs.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.739194Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/openebs.io/v1beta2, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.740830Z DEBUG kube_client::discovery::apigroup: Listing group versions, name: "metrics.k8s.io"
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/discovery/apigroup.rs:80

  2024-08-19T19:40:59.740939Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/metrics.k8s.io/v1beta1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.745818Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.747540Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1, otel.name: "HTTP", otel.kind: "client"

  2024-08-19T19:40:59.751085Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/apis/snapshot.storage.k8s.io/v1/volumesnapshotcontents?&fieldSelector=&labelSelector=&limit=2, otel.name: "list", otel.kind: "client"

     Collecting Kuberbetes pod resources
  2024-08-19T19:40:59.767874Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/pods?&fieldSelector=&labelSelector=&limit=100, otel.name: "list", otel.kind: "client"

Completed collection of Kubernetes resource specific information
Collecting mayastor specific information from Etcd...
  2024-08-19T19:40:59.814977Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.250.82:16443/api/v1/namespaces/mayastor/pods?&fieldSelector=status.phase%3DRunning&labelSelector=app%3Detcd, otel.name: "list", otel.kind: "client"

  2024-08-19T19:40:59.847041Z ERROR kube_forward::port_forward: failed to forward connection, error: No pod 'app=etcd' available
    at dependencies/control-plane/k8s/forward/src/port_forward.rs:81

Failed to collect etcd dump information, error: Etcd(GetPrefix { prefix: "/openebs.io/mayastor/apis/v0/clusters/b3ab0f6d-300d-47f6-837f-30ffd08418aa/namespaces/mayastor", source: GRpcStatus(Status { code: Unknown, message: "transport error", source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Kind(ConnectionReset)))) }) })
Completed collection of dump !!
core@sc-os-80-node2:~$ ls
mayastor-2024-08-19--19-12-06-UTC.tar.gz  mayastor-2024-08-19--19-40-59-UTC.tar.gz  snap  verisign-enabled
core@sc-os-80-node2:~$ ls -al
total 624
drwxr-x--- 7 core core   4096 Aug 19 12:40 .
drwxr-xr-x 4 root root   4096 Jul 31 10:04 ..
drwx------ 3 core core   4096 Jul 31 10:37 .ansible
-rw------- 1 core core     96 Aug  6 14:16 .bash_history
-rw-r--r-- 1 core core    220 Jan  6  2022 .bash_logout
-rw-r--r-- 1 core core   3771 Jan  6  2022 .bashrc
drwx------ 2 core core   4096 Jul 31 10:37 .cache
drwxrwxrwx 3 root root   4096 Aug 19 12:10 .kube
-rw-rw-r-- 1 core core 552761 Aug 19 12:14 mayastor-2024-08-19--19-12-06-UTC.tar.gz
-rw-rw-r-- 1 core core  38594 Aug 19 12:40 mayastor-2024-08-19--19-40-59-UTC.tar.gz
-rw-r--r-- 1 core core    807 Jan  6  2022 .profile
drwxrwxrwx 2 root root   4096 Jul 31 11:15 snap
drwx------ 2 core core   4096 Jul 31 10:22 .ssh
-rw-r--r-- 1 core core      0 Jul 31 10:20 .sudo_as_admin_successful
-rw-r--r-- 1 root root      0 Jul 31 10:20 verisign-enabled
core@sc-os-80-node2:~$ tar -tvf mayastor-2024-08-19--19-40-59-UTC.tar.gz
drwxrwxr-x 1000/1000         0 2024-08-19 12:40 ./
-rw-rw-r-- 1000/1000         0 2024-08-19 12:40 etcd_dump
drwxrwxr-x 1000/1000         0 2024-08-19 12:40 k8s_resources
-rw-rw-r-- 1000/1000    264705 2024-08-19 12:40 k8s_resources/pods.yaml
-rw-rw-r-- 1000/1000    118686 2024-08-19 12:40 k8s_resources/k8s_events.json
-rw-rw-r-- 1000/1000         3 2024-08-19 12:40 k8s_resources/volume_snapshot_contents.yaml
drwxrwxr-x 1000/1000         0 2024-08-19 12:40 k8s_resources/configurations
-rw-rw-r-- 1000/1000     12560 2024-08-19 12:40 k8s_resources/configurations/mayastor-io-engine.yaml
-rw-rw-r-- 1000/1000      8531 2024-08-19 12:40 k8s_resources/configurations/mayastor-agent-ha-node.yaml
-rw-rw-r-- 1000/1000     11049 2024-08-19 12:40 k8s_resources/configurations/mayastor-csi-node.yaml
-rw-rw-r-- 1000/1000      9911 2024-08-19 12:40 k8s_resources/configurations/mayastor-agent-core.yaml
-rw-rw-r-- 1000/1000      7757 2024-08-19 12:40 k8s_resources/configurations/mayastor-localpv-provisioner.yaml
-rw-rw-r-- 1000/1000     12459 2024-08-19 12:40 k8s_resources/configurations/mayastor-csi-controller.yaml
-rw-rw-r-- 1000/1000     14936 2024-08-19 12:40 k8s_resources/configurations/mayastor-etcd.yaml
-rw-rw-r-- 1000/1000      6840 2024-08-19 12:40 k8s_resources/configurations/mayastor-operator-diskpool.yaml
-rw-rw-r-- 1000/1000     12104 2024-08-19 12:40 k8s_resources/configurations/mayastor-nats.yaml
-rw-rw-r-- 1000/1000      7982 2024-08-19 12:40 k8s_resources/configurations/mayastor-api-rest.yaml
-rw-rw-r-- 1000/1000      1704 2024-08-19 12:40 k8s_resources/k8s_disk_pools.yaml
-rw-rw-r-- 1000/1000         3 2024-08-19 12:40 k8s_resources/volume_snapshot_classes.yaml
-rw-rw-r-- 1000/1000      1099 2024-08-19 12:40 support_tool_logs.log
veenadong commented 2 months ago

Update on the mayastor plug-in returning an error:

core@glop-nm-126-mem3:~$ RUST_LOG=debug kubectl-mayastor get volumes
  2024-08-20T22:47:49.773676Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.244.130:16443/api/v1/namespaces/mayastor/services?&labelSelector=app%3Dapi-rest, otel.name: "list", otel.kind: "client"

  2024-08-20T22:47:49.792212Z  INFO kube_forward::http_forward: generated kube-api, uri: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy
    at dependencies/control-plane/k8s/forward/src/http_forward.rs:54

  2024-08-20T22:47:49.847900Z DEBUG kube_client::client::builder: requesting
    at /sources/kube-client-0.85.0-98989b6e1f27695afe22aa29c94136fa06be5e8d28b91222e6dfbe5a460c803f/src/client/builder.rs:127
    in kube_client::client::builder::HTTP with http.method: GET, http.url: https://10.245.244.130:16443/api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: "HTTP", otel.kind: "client"
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client"

  2024-08-20T22:47:59.848699Z DEBUG openapi::clients::tower::configuration: failed to issue request, error: request timed out
    at dependencies/control-plane/openapi/src/clients/tower/configuration.rs:241
    in openapi::clients::tower::configuration::HTTP with http.method: GET, http.url: /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.name: GET /api/v1/namespaces/mayastor/services/http:mayastor-api-rest:http/proxy/v0/volumes?max_entries=200&starting_token=0, otel.kind: "client", otel.status_code: "ERROR"

Failed to list volumes. Error error in request: request timed out

List of running pods:

core@glop-nm-126-mem3:~$ kubectl get pods -n mayastor
NAME                                            READY   STATUS    RESTARTS      AGE
mayastor-agent-core-88bc8d8b9-8vk97             2/2     Running   0             46m
mayastor-agent-ha-node-vr9mb                    1/1     Running   5 (20h ago)   20h
mayastor-agent-ha-node-z5zgh                    1/1     Running   0             121m
mayastor-api-rest-d666fb647-6z7w7               1/1     Running   0             5h52m
mayastor-api-rest-d666fb647-9dv5m               1/1     Running   0             19h
mayastor-api-rest-d666fb647-fm97p               0/1     Pending   0             46m
mayastor-csi-controller-799f88d74d-hqtct        6/6     Running   0             5h46m
mayastor-csi-node-dn4sk                         2/2     Running   0             121m
mayastor-csi-node-txhvn                         2/2     Running   0             151m
mayastor-etcd-0                                 1/1     Running   0             5h46m
mayastor-etcd-1                                 1/1     Running   0             19h
mayastor-etcd-2                                 0/1     Pending   0             40m
mayastor-io-engine-2k44l                        2/2     Running   0             121m
mayastor-io-engine-s5l92                        2/2     Running   0             156m
mayastor-localpv-provisioner-796cdf4d5b-gpcn5   1/1     Running   0             41m
mayastor-nats-0                                 3/3     Running   0             5h46m
mayastor-nats-1                                 3/3     Running   0             40m
mayastor-nats-2                                 3/3     Running   0             40m
mayastor-operator-diskpool-5c894c69bf-nwjhj     1/1     Running   0             41m

Attaching agent-core logs: mayastor-agent-core-88bc8d8b9-8vk97.log.gz

mayastor-api-rest-d666fb647-9dv5m.log.gz mayastor-api-rest-d666fb647-6z7w7.log.gz

tiagolobocastro commented 2 months ago

You seem to have configured 3 replicas for the api-rest, any reason for this? Seems that one of them is in Pending:

mayastor-api-rest-d666fb647-6z7w7               1/1     Running   0             5h52m
mayastor-api-rest-d666fb647-9dv5m               1/1     Running   0             19h
mayastor-api-rest-d666fb647-fm97p               0/1     Pending   0             46m

Also etcd has a pending pod:

mayastor-etcd-0                                 1/1     Running   0             5h46m
mayastor-etcd-1                                 1/1     Running   0             19h
mayastor-etcd-2                                 0/1     Pending   0             40m

Anyway, moving on to the error, is due to timeout:

  2024-08-20T22:48:17.698942Z ERROR grpc::operations::volume::client: error: Aborted: status: Cancelled, message: "Timeout expired", details: [], metadata: MetadataMap { headers: {} }
    at control-plane/grpc/src/operations/volume/client.rs:76

  2024-08-20T22:48:46.785375Z  INFO actix_web::middleware::logger: 172.23.101.192 "GET /v0/volumes?max_entries=200&starting_token=0 HTTP/1.1" 503 143 "-" "-" 54.687851
    at /build/cargo-vendor-dir/actix-web-4.4.0/src/middleware/logger.rs:420

But on the agents-core there's nothing there which would explain the timeout... I can see requests taking minutes to complete though:

  2024-08-20T22:51:27.585545Z  INFO actix_web::middleware::logger: 172.23.138.198 "GET /v0/nodes/glop-nm-126-mem3.glcpdev.cloud.hpe.com/pools/glop-nm-126-mem3.glcpdev.cloud.hpe.com-disk HTTP/1.1" 503 143 "-" "-" 144.404435
    at /build/cargo-vendor-dir/actix-web-4.4.0/src/middleware/logger.rs:420

  2024-08-20T22:51:27.585601Z  INFO actix_web::middleware::logger: 172.23.138.198 "GET /v0/nodes/glop-nm-126-mem3.glcpdev.cloud.hpe.com/pools/glop-nm-126-mem3.glcpdev.cloud.hpe.com-disk HTTP/1.1" 503 143 "-" "-" 49.604255
    at /build/cargo-vendor-dir/actix-web-4.4.0/src/middleware/logger.rs:420

@Abhinandan-Purkait any ideas here? Maybe we should enable tracing?

Abhinandan-Purkait commented 2 months ago

@Abhinandan-Purkait any ideas here?

I think it's because the rest-api isn't supposed to run with multiple instances, as it doesn't have any sort of loadbalancing and or leader/slave mechanism. Timeouts probably because the same request is being served by different instances of rest-api and the locks held at core-agent puts one on wait?

I believe it has nothing to do with the connectivity or computing time due to load?

tiagolobocastro commented 2 months ago

@veenadong could you please change agents.core.logLevel to debug? This should help provide more information. If it doesn't we can try enabling jaeger to get more detailed information

Abhinandan-Purkait commented 2 months ago

@Abhinandan-Purkait any ideas here?

I think it's because the rest-api isn't supposed to run with multiple instances, as it doesn't have any sort of loadbalancing and or leader/slave mechanism. Timeouts probably because the same request is being served by different instances of rest-api and the locks held at core-agent puts one on wait?

I believe it has nothing to do with the connectivity or computing time due to load?

I just checked and I think the above could not be, as in a Kubernetes ClusterIP service setup, it is highly unlikely for two different pods to receive the exact same request under normal circumstances, as it kind of does load balancing by default, round robin.

Yeah, it would be good to see debug logs.