NFS PV allocation is not dynamic - use nfs-provisioner

geerlingguy / raspberry-pi-dramble

DEPRECATED - Raspberry Pi Kubernetes cluster that runs HA/HP Drupal 8

http://www.pidramble.com/

MIT License

1.67k stars 260 forks source link

NFS PV allocation is not dynamic - use nfs-provisioner #134

Closed geerlingguy closed 5 years ago

geerlingguy commented 5 years ago

See: https://github.com/kubernetes-incubator/external-storage/tree/master/nfs

Basically, some of the PVCs are not binding because only one PV is getting created, and it is being used by other PVCs in other namespaces:

Events:
  Type       Reason         Age                    From                         Message
  ----       ------         ----                   ----                         -------
  Normal     FailedBinding  3m48s (x222 over 59m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
Mounted By:  docker-registry-6c556fc864-dnrk9

I was under the impression I could have many PVs on one NFS mount, and then PVCs that mount on that PV. But apparently not.

I might be able to do localstorage instead, and just use the NFS mount path and a hardcoded directory... but it would be nicer to have NFS PVs dynamically allocated and managed (easier to make sure resources are cleaned up, e.g. when deleting a namespace).

geerlingguy commented 5 years ago

So, I was initially implementing the nfs provisioner, which runs its own NFS server internally in the cluster, tied to a hostPath (or any other kind of volume you can attach to a pod) in the cluster.

What I want to do, though, is use nfs-client, which uses an already-existing/already-set-up NFS server that the cluster can see.

The main point of this is so that I can have the contents of the NFS server mounted and visible on all the nodes. The way both are configured, you necessarily tie (at least in this infrastructure) the NFS share to one particular server (so pet over cattle), and with nfs-client, that NFS share/server is managed externally (in this case by Ansible), and with nfs, that share/server is managed by the deployment running inside the cluster.

There are advantages and tradeoffs either way, but since I already have NFS configured external from the Kubernetes cluster itself, I'll keep it that way for now at least.

geerlingguy commented 5 years ago

Running into the following issues:

Looks like an RBAC permissions issue currently. Trying a fix.

geerlingguy commented 5 years ago

🤦‍♂️

I still had the old nfs RBAC manifest applying for nfs-client... once I actually started using the RBAC that comes with nfs-client, it started magically working. It's amazing how things work if you just use the right manifest :)

geerlingguy commented 5 years ago

Commit above should work, but will still fail Travis test due to an issue with #132.

geerlingguy commented 5 years ago

Testing on the Pi cluster currently. Had to rebuild it again... that's like 8 times this weekend :D