[RFE] Add Persistent Storage Providers

[usrbinkat]

Add Storage Providers

Requirements

1. A simple and reliable mvp storage provider that can be simply toggled on/off via bool in pulumi config and requires minimal/no user configuration.
2. A highly configurable and available storage provider supporting sane hdd, ssd, nvme storage class device matching and also serve the platform undercloud block devices as storage classes to tenant workloads and tenant kubernetes clusters as well.
3. A NFS storage provisioner for easy long term persistent storage served from appliance or provisioned NAS.

Selection

1. [x] rancher hostpath provisioner
   • example pulumi iac usrbinkat/homelab-apps/ops/local-path-provisioner
2. [x] rook ceph operator
   • last known good rook helm values rook ceph values
3. [ ] rook ceph cluster(s)
   • last known good rook helm values rook ceph values
4. [ ] nfs democratic-csi provisioner
   • example helm values usrbinkat/homelab-apps/ops/democratic-csi

[MichaelCade]

Adding for reference rook-ceph - https://www.talos.dev/v1.6/kubernetes-guides/configuration/ceph-with-rook/

[MichaelCade]

reference for PSP stopped me from progressing.

       admissionControl:
            - name: PodSecurity # Name is the name of the admission controller.
              configuration:
                apiVersion: pod-security.admission.config.k8s.io/v1alpha1
                defaults:
                    audit: restricted
                    audit-version: latest
                    enforce: baseline
                    enforce-version: latest
                    warn: restricted
                    warn-version: latest
                exemptions:
                    namespaces:
                        - kube-system
                        - rook-ceph