vcluster on OpenShift 4

[tobru]

I'd like to get vcluster running on OpenShift and hit the well-known security constraints of OpenShift.

With the following values.yaml I got the vcluster Pod up and running:

volumes:
- name: data
  emptyDir: {}
- name: k3s-config
  emptyDir: {}
- name: kubeconfig
  emptyDir: {}
- name: roothome
  emptyDir: {}
- name: vclusterdata
  emptyDir: {}
vcluster:
  volumeMounts:
  - mountPath: /data
    name: data
  - mountPath: /k3s-config
    name: k3s-config
  - mountPath: /.kube
    name: kubeconfig
syncer:
  volumeMounts:
  - mountPath: /data
    name: data
  - mountPath: /.kube
    name: kubeconfig
  - mountPath: /root
    name: roothome
  - mountPath: /var/lib/vcluster
    name: vclusterdata

This solves all the filesystem permission constraints. It's not perfect (e.g. no persistence), but enough for a PoC to at least get the vcluster Pod running.

The issue I now face is that the CoreDNS Pod which is synced from the vcluster to the host cluster isn't able to start. I get the following message in the syncer container log:

E1029 14:47:09.744026       1 controller.go:302] controller-runtime: manager: reconciler group  reconciler kind Pod: controller: pod-forward: name coredns-7448499f4d-rzmd7 namespace kube-system: Reconciler error pods "coredns-7448499f4d-rzmd7-x-kube-system-x-vcluster-1" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.capabilities.add: Invalid value: "NET_BIND_SERVICE": capability may not be added, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "syn-cluster-backup-etcd": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

That means I have to probably reconfigure the CoreDNS Pod which is started in the vcluster context. Any hints how I can this up and running?

[FabianKramm]

@tobru thanks a lot for this issue! Yeah we are working on a non root startup mode for vcluster, but coredns capabilities seem to be a problem in your setup. Does it work if you remove the security context configuration from the coredns deployment?

[tobru]

When I simply remove .spec.containers[0].securityContext.capabilities from the container, CoreDNS can't start:

Listen: listen tcp :53: bind: permission denied

So the CoreDNS configuration needs some refinement for OpenShift. I'll see what I can come up with.

[tobru]

By changing the CoreDNS config to listen e.g. on 8053 and changing the Service and Pod configuration accordingly the CoreDNS Pod starts without errors. I also had to remove the NodeHosts mount and removed this part from the CoreDNS config.

This made it run, but I'm unsure about some points:

• Why does the original CoreDNS configuration and deployment of k3s bind to port 53 (and therefore needs to set the NET_BIND_SERVICE capability)? I could imagine that this allows to resolve names from the hosts. Is this needed in vcluster?
• The NodeHosts ConfigMaps seems to only be there if you let k3s manage CoreDNS, see https://github.com/k3s-io/k3s/issues/1328. Maybe this isn't needed for vcluster anyway?

All-in-all I think for getting it to run on OpenShift (and other distributions with a very restricted enforcement) the CoreDNS deployment needs to be customized. Any thoughts on bringing that into vcluster? I might be able to do some patchy stuff to still get an automated vcluster provisioning, but I'd rather see this part of vcluster.

[matskiv]

@tobru Thank you for sharing your findings, they are helpful. I am now looking at storage and CoreDNS configuration for environments with security constraints, and we plan to bring this into vcluster.