search

ccremer / kubernetes-zfs-provisioner

Dynamic ZFS persistent volume provisioner for Kubernetes
Apache License 2.0
74 stars 7 forks source link

Error creating lock: endpoints lock is removed, migrate to endpointsleases #70

Closed developStorm closed 2 years ago

developStorm commented 2 years ago

Deployment fails with error message in title.

I believe this is an upstream issue: https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/issues/123. As mentioned in the linked issue, the problem was actually fixed in upstream but a release with that fix was never made. This project is using lateset release, v8, of storage-lib-external-provisioner which has the problem.

I took a look at the nfs-subdir-external-provisioner to see how they managed to mitigate this issue, and turns out they limited the version of storage-lib-external-provisioner to v6, which doesn't have the problematic line of code. I think we can probably do the same before the upstream release the fix to at lease make this project usable?

ccremer commented 2 years ago

Hi. Thanks for reporting the issue. If there's a fix upstream but never released, I'd rather just use the latest commit in master than reverting to v6 and then later upgrade to v8 again after a release.

ccremer commented 2 years ago

I've release v1.2.0 as a hotfix, since I didn't test it myself. If the problem persists, please reopen the issue.

developStorm commented 2 years ago

Hi @ccremer thanks for the quick fix! The hotfix seems to fix the original issue but I'm experiencing a new permission error now:

[leaderelection.go:248] attempting to acquire leader lease zfs/pv.kubernetes.io-zfs... [leaderelection.go:330] error retrieving resource lock zfs/pv.kubernetes.io-zfs: leases.coordination.k8s.io "pv.kubernetes.io-zfs" is forbidden: User "system:serviceaccount:zfs:kubernetes-zfs-provisioner" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "zfs"

I have rbac: true in my values file, so maybe this error is introduced by the fix...(fix introduced new permission needs)?

developStorm commented 2 years ago

Applying this seems fixed it for me:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubernetes-zfs-provisioner-lease
  namespace: zfs
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update", "patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubernetes-zfs-provisioner-lease
  namespace: zfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-zfs-provisioner-lease
subjects:
- kind: ServiceAccount
  name: kubernetes-zfs-provisioner
  namespace: zfs
ccremer commented 2 years ago

Jup, that's what I did just here: https://github.com/ccremer/kubernetes-zfs-provisioner/releases/tag/kubernetes-zfs-provisioner-2.0.1, feel free to try it out