Expose Kubernetes API Server MagicDNS w/ Egress + SSL

saadbahir commented 3 months ago

What is the issue?

Hello everyone,

Context

I have the following use case:

1 k8s cluster (Cluster 1) with ArgoCD
a 2nd k8s cluster (Cluster 2)

Goal:

I want to make my clusters endpoint private and use exclusively tailscale for the connection to the clusters
In my use case, I specifically want my ArgoCD deployment to connect to the 2nd k8s cluster through tailnet

kubectl exec busybox -- nslookup my-cluster.tailscale-domain.ts.net
nslookup: can't resolve 'my-cluster.tailscale-domain.ts.net'

Problem:

Cluster 1 cannot connect to Cluster 2 using MagicDNS
. I could try to setup tailscale on every node but 1/ it's very cumbersome and unpracticable 2/ I use AWS EC2 which is not suitable for enabling MagicDNS
I tried using Egress on Cluster 1 to expose my Cluster 2 API server proxy MagicDNS but it can only support HTTP => I want to keep the SSL validation on my cluster (for obvious security reasons)

What I have done so far:

I deployed kubernetes operator in both clusters
I deployed subnet router over both clusters
I can reach BOTH clusters through my local environment over tailnet
I still couldn't reach one cluster from the other

I also tried editing coredns configmap following this recommendation to add the magicDNS nameserver

Maybe I misunderstand the premise of the kubernetes operator and my use case is not covered

Has anyone tried to fix this use case? If so, what can I try to resolve it?

Thank you for your help!

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

No response

Tailscale version

No response

Other software

No response

Bug report

No response

irbekrm commented 3 months ago

Hi @saadbahir

Thank you for creating the issue and for the use case description.

We actually hadn't thought about cross-cluster access to the kube-apiserver via the cluster MagicDNS, but I think that's a really good use case!

I just tested it and it worked for me.

As a note, since #10499 has not merged yet and also we haven't yet published official nameserver images, this functionality is a bit difficult to test.

The way I tested it was:

deploy Kubernetes operator in api-server proxy mode in cluster 2 (as from your example) to expose the API server in that cluster
deploy Kubernetes operator in cluster 1
create DNSConfig and modify kube-dns in cluster 1 as per the PR description in https://github.com/tailscale/tailscale/pull/11019 (at the moment there isn't a very straightforward way to test this, you would have to use gcr.io/csi-test-290908/k8s-nameserver:v0.0.4crossclusterapi as the nameserver image or build your own- we haven't pushed images yetO

I then created an egress Service in cluster 1 pointing at the operator MagicDNS name in cluster 2:

apiVersion: v1
kind: Service
metadata:
annotations:
tailscale.com/tailnet-fqdn: <full-operator-magic-dns-name-from-cluster-2>
name: operator-egress
namespace: default
spec:
externalName: placeholder
type: ExternalName

in cluster 2 I had to configure RBAC to allow the tag of the egress proxy Kubernetes permissions
I then tested this in cluster 1 by installing kubectl to a non-tailscale workload Pod and adding there kubeconfig generated by tailscale configure kubeconfig. This worked and I was able to run kubectl commands for against the Kubernetes APi server in cluster 1 from a workload Pod in cluster 2.

We will document these steps better once all of the work for the MagicDNS name resolution in cluster gets merged and we have published the nameserver images.

Keen to hear if the this will work for you and also especially if the last steps (configuring RBAC for proxy tags in cluster 2 and passing kubeconfig to cluster workloads) will make sense for your workflow.

I am not very familiar with cross-cluster ArgoCD- is there a way to pass cluster 2 kubeconfig to it, or does it need to be configured in some other way?

sauyon commented 3 months ago

I believe the easy workaround I found for specifically argocd was just setting this:

tls_client_config:
  server_name: <magicdns name>

Then you can simply use the cluster URL for access.

tailscale / tailscale