Volumes are not assigned to a node

[Fersoil]

What steps did you take and what happened: I have a kubernetes cluster with 3 worker nodes running openebs zfs driver. I have recently noticed that my volumes are not assigned to one specific node called mickey. After forcing kubernetes to create pod on that specific node it shows error 1 node(s) did not have enough free storage.

Every service needed seems to be running on the nodes and pools are healthy. I can create filesystems, etc. manually on the node and there are a lot of free storage in the pool.

What did you expect to happen: I would expect to pvs be assigned into the mickey node.

The output of the following commands will help us better understand what's going on:

• kubectl get pods -n kube-system
• https://gist.github.com/Fersoil/0ca13773625d96a9e4a700d484fa1231
• kubectl get zv -A -o yaml https://gist.github.com/Fersoil/43bcbd6c349085493c4b376581c85c76

Anything else you would like to add: Everything worked just fine and there are no recent errors

Environment:

• ZFS-LocalPV version zfs-2.1.5-1ubuntu6~22.04.1 zfs-kmod-2.1.5-1ubuntu6~22.04.1
• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:09:57Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version: kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:36:57Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: the cluster is bare metal installed using kubeadm
• OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS

[sinhaashish]

Can you please check PR https://github.com/openebs/zfs-localpv/pull/451 .

[avishnu]

Can you share the ZfsNode CRs and the logs? And what's the volume binding mode for the PVs?