Need the ability to specify the equivalent of root_cas: host in the ingress controller

[ctindel]

Kubernetes Version

latest

Helm Chart Version

latest

Helm Chart configuration

No response

What happened

When using a custom serverAddr like --set serverAddr="tunnel.us.connect.example.com:443" if that edge domain was created with a cert signed by letsencrypt, we need the ability to tell the ingress controller to trust the host's root CA store just like we do when using that custom serverAddr with the CLI agent.

We get this error:

{"level":"error","ts":"2024-04-29T17:25:43Z","logger":"controllers.tunnel","msg":"Reconciler error","reconcileID":"5aebdabc-0fef-4fbc-94c7-b72b59cdf063","error":"failed to send authentication request: tls: failed to verify certificate: x509: certificate signed by unknown authority","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}

What you think should happen instead

No response

How to reproduce

No response

[bobzilladev]

• WithCA is called here: https://github.com/ngrok/kubernetes-ingress-controller/blob/main/pkg/tunneldriver/driver.go#L106
• It checks dir /etc/ssl/certs/ngrok/: https://github.com/ngrok/kubernetes-ingress-controller/blob/main/pkg/tunneldriver/driver.go#L44
• Which "combines the system ca certs with a directory of custom ca certs" here: https://github.com/ngrok/kubernetes-ingress-controller/blob/main/pkg/tunneldriver/driver.go#L196
• There's a comment about allowing that dir to be overrideable via helm config, that is one possibility
• To make this work for a customer, having them put the certs needed in /etc/ssl/certs/ngrok/, or even just making sure that directory exists, may get the job done

[jrobsonchase]

Oh, neat. I didn't see this comment and apparently didn't actually read very much of the tunneldriver code I was messing with. Sounds like that solves the problem assuming the ssl certs directory isn't too hard to create.

[bobzilladev]

Still makes sense to have a setting like your PR to not need to deal with directories, but could be a quick-fix with existing release at least

[ctindel]

@jrobsonchase ideally we would make it more like the agent so the user doesn't have to change anything. While it may be possible to do, it creates an overhead with extra steps for making this work especially in managed k8s services like gke, eks etc where the user likely isn't customizing anything else on the worker nodes already.

[jrobsonchase]

ideally we would make it more like the agent so the user doesn't have to change anything

Is this in reference to the directory creation or adding --set hostCA=true to the helm arguments?

Unless I missed something (which is entirely possible), we still require you to set root_cas in addition to the server_addr in the agent config, so the helm argument is right in line with how the agent works.

[ctindel]

@jrobsonchase What I meant was I think it's better to have a --set hostCA=true since we can universalize those instructions, as opposed to telling people how to go into different systems like k3d, EKS, AKS, GKE and muck around with the local filesystem.