Open veenadong opened 1 month ago
What is the value of repl in the storage class? Did the node come back online?
Seems you have hit this bug: https://github.com/openebs/mayastor/issues/1710
[2024-08-08T01:16:25.585879277+00:00 ERROR mayastor::spdk:nvme_fabric.c:596] Connect command failed, rc -125, trtype:TCP adrfam:IPv4 traddr:10.245.250.71 trsvcid:8420 subnqn:nqn.2019-05.io.openebs:16b7a8d2-e89e-4e31-9e8f-571a3726a986 [2024-08-08T01:16:25.585921213+00:00 ERROR mayastor::spdk:nvme_tcp.c:2017] Failed to poll NVMe-oF Fabric CONNECT command [2024-08-08T01:16:25.585926825+00:00 ERROR mayastor::spdk:nvme_tcp.c:1806] Failed to connect tqpair=0x5582015e9850
We have a fix on https://github.com/openebs/mayastor/pull/1711
hmm no logs are included in the bundle @veenadong, which plugin version did you use to create the bundle?
hmm no logs are included in the bundle @veenadong, which plugin version did you use to create the bundle?
We are running 2.5.1 and the plugin version is:
Kubectl Plugin (kubectl-mayastor) revision e8f685e3c9e1 (v2.5.0+0)
I see the fix #1711 will be merged into 2.7? What is the ETA for this fix to be available for GA?
Plan is to release in a couple of weeks along with a control-plane previously missed functionality for updating pool labels after creation.
Mayastor 2.5.1 on a 3 node system, take 1 node down and many volumes are in a rebuilding then faulting, and trying again.
mayastor-io-engine-xs9sk_mayastor_io-engine-dd23276f2aaa3971af97201157038ba42b9e57962f307d92e693116527acd457.log:2024-08-07T18:19:38.950624012-07:00 stdout F [2024-08-08T01:19:38.950553630+00:00 ERROR io_engine::rebuild::rebuild_descriptor:rebuild_descriptor.rs:85] 10.245.250.69:8420/nqn.2019-05.io.openebs:b8d8cb2e-0160-4015-931a-53c3b125447cn1: failed to get I/O handle: failed to get IO channel for 10.245.250.69:8420/nqn.2019-05.io.openebs:b8d8cb2e-0160-4015-931a-53c3b125447cn1
Mayastor pods are running:IO-engine pod is logging:
Attached are the support bundle from the two running nodes: mayastor-2024-08-08--00-35-44-UTC.tar.gz mayastor-2024-08-08--00-34-37-UTC.tar.gz