Generating secrets for additional static pods

[maxpain]

Hello.

I want to deploy Konnectivity server as a StaticPod.

apiVersion: v1
kind: Pod
metadata:
  name: konnectivity-server
  namespace: kube-system
spec:
  priorityClassName: system-cluster-critical
  hostNetwork: true
  containers:
  - name: konnectivity-server-container
    image: registry.k8s.io/kas-network-proxy/proxy-server:v0.0.37
    command: ["/proxy-server"]
    args: [
            "--logtostderr=true",
            # This needs to be consistent with the value set in egressSelectorConfiguration.
            "--uds-name=/etc/kubernetes/konnectivity-server/konnectivity-server.socket",
            "--delete-existing-uds-file",
            # The following two lines assume the Konnectivity server is
            # deployed on the same machine as the apiserver, and the certs and
            # key of the API Server are at the specified location.
            "--cluster-cert=/etc/kubernetes/pki/apiserver.crt",
            "--cluster-key=/etc/kubernetes/pki/apiserver.key",
            # This needs to be consistent with the value set in egressSelectorConfiguration.
            "--mode=grpc",
            "--server-port=0",
            "--agent-port=8132",
            "--admin-port=8133",
            "--health-port=8134",
            "--agent-namespace=kube-system",
            "--agent-service-account=konnectivity-agent",
            "--kubeconfig=/etc/kubernetes/konnectivity-server.conf",
            "--authentication-audience=system:konnectivity-server"
            ]
    livenessProbe:
      httpGet:
        scheme: HTTP
        host: 127.0.0.1
        port: 8134
        path: /healthz
      initialDelaySeconds: 30
      timeoutSeconds: 60
    ports:
    - name: agentport
      containerPort: 8132
      hostPort: 8132
    - name: adminport
      containerPort: 8133
      hostPort: 8133
    - name: healthport
      containerPort: 8134
      hostPort: 8134
    volumeMounts:
    - name: k8s-certs
      mountPath: /etc/kubernetes/pki
      readOnly: true
    - name: kubeconfig
      mountPath: /etc/kubernetes/konnectivity-server.conf
      readOnly: true
    - name: konnectivity-uds
      mountPath: /etc/kubernetes/konnectivity-server
      readOnly: false
  volumes:
  - name: k8s-certs
    hostPath:
      path: /etc/kubernetes/pki
  - name: kubeconfig
    hostPath:
      path: /etc/kubernetes/konnectivity-server.conf
      type: FileOrCreate
  - name: konnectivity-uds
    hostPath:
      path: /etc/kubernetes/konnectivity-server
      type: DirectoryOrCreate

But I don't know the proper way to generate and provide secrets such as:

• --kubeconfig
• --cluster-cert
• --cluster-key

Does Talos provide some way to generate such secrets for static pods?

[smira]

There is no supported way at the moment that would be good enough.

A workaround is to use .machine.files to lay down secrets and mount them up into a static pod as volume mounts.

There are files generated for controlplane components under /system/secrets which might work, but using them might break in the fturue.

Probably given the nature of the app, can it be ran as a DaemonSet on controlplanes instead?

[maxpain]

Probably given the nature of the app, can it be ran as a DaemonSet on controlplanes instead?

Yes, but I want to hide the control plane nodes using skipNodeRegistration so I have to run Konnectivity as a static pod. I'm also looking for other not-so-hacky solutions https://github.com/siderolabs/talos/issues/9395

[smira]

A proper way would be for Talos to support such secrets explicitly, that is in the static pod specify that you want a Kubernetes CA, and Talos would manage that automatically.

[smira]

We will look into #9395, but no commitment yet.