k8s, code/hiera deployment isn't working because of ReadWriteOnce volume

[baurmatt]

Describe the Bug

The k8s Helm Chart uses a PersistentVolume (puppet-code-claim) for the Puppet code (/etc/puppetlabs/code/) storage. This gets initialized by a PersistentVolumeClaim as ReadWriteOnce.

The PVC (puppet-code-claim) then gets used by the Puppetserver Deployment and both r10k Cronjobs (code/hiera). Due to the fact that the volume being a ReadWriteOnce, this doesn't work.

This effectively leads to broken setup as the code will never be deployed.

Expected Behavior

Quick and dirty: s,ReadWriteOnce,ReadWriteMany,g

But because ReadWriteMany isn't support by a many cloud providers, I would prefer a solution which doesn't depend on the cloud provider offering a ReadWriteMany.

My current approach would be a webhook based version instead of cronjobs:

• Use emptyDir volumes instead of a PVC for code/hiera
• Every Puppetserver pod gets a sidecar which runs a webhook which can trigger a local r10k redeployment of code/hiera
• Create a central webhook receiver pod which can be used to push Github/Gitlab/... webhook to it. This receiver should forward the redeploy request to all currently running Puppetserver pods.

Steps to Reproduce

Deploy the helm chart on a multi node cluster (not multi az/dc! ;))

Environment

Additional Context

Add any other context about the problem here.

[baurmatt]

Another alternative would be to run the cronjob in the sidecar.

[baurmatt]

Moaaar options :D https://github.com/kubernetes/git-sync

[Xtigyro]

@baurmatt Hey - thanks for reporting it.

Right now we don't support running multiple Puppet masters on different nodes which sit in different cloud zones but we have plans and work has started on that.

In the README you can find information on how to setup properly Puppet Server in the cloud using a K8s Storage Class here, how to set pod affinity for r10k here or setting common node selectors for Puppet Server and r10k here.

Have you tested those options?