FR: Kubernetes operator: make it possible to resolve Tailnet FQDNs from within the Kubernetes cluster

irbekrm commented 9 months ago

What are you trying to do?

Today, tailnet services can be exposed to Kubernetes cluster workloads using cluster egress proxies via the Tailscale Operator. Once an egress proxy has been set up in a Kubernetes cluster, any cluster workload can access the tailnet service by referring to it via the DNS name of the Kubernetes Service that we create for the egress proxy.

However, in some cases, it might be required that the workload can refer to the tailnet service using its MagicDNS name There are two known use cases, a combination of both is possible:

For a cluster workload to access a tailnet service over TLS (assuming Tailscale HTTPS) the workload needs to use the MagicDNS name (because that's what the Tailscale HTTPS cert is issued for).
In some cases multiple components, some of which might be running in cluster and some might be user devices on tailnet, need to be configured to use the same URL to access a service, see https://github.com/tailscale/tailscale/issues/10027#issuecomment-1827763853

How should we solve this?

A possible solution was implemented in https://github.com/tailscale/tailscale/pull/9201

We could:

deploy a simple nameserver in the cluster
when a Tailscale operator creates an egress proxy it could also update the nameserver with an A record for the MagicDNS name to the IP address of a Kubernetes Service that points to the egress proxy

Users would have to:

update their CoreDNS/KubeDNS config with a stub nameserver record for their tailnet's subdomain to the nameserver that we deployed.

Any cluster workload would then be able to access the exposed tailnet service via MagicDNS name.

What is the impact of not solving this?

Folks who need to access access tailnet services over HTTPS would need to manually configure DNS to make this possible. Folks who need to access workloads exposed via tailscale ingress from within the cluster, might not be able to do this at all.

Anything else?

No response

irbekrm commented 7 months ago

Currently this is being worked on in #11017 , #11019. Here is an approximate example user flow for accessing ingress proxy from within cluster that will be possible once the above linked work is done:

A user wants to expose a kube workload to tailnet over HTTPS and refer to it by the same hostname (the MagicDNS name of the ingress proxy) from tailnet as well as from the cluster:

Assume that CoreDNS is used as the cluster DNS solution.

Deploy the operator as usual.
Create a new DNSConfig custom resource that tells the operator to create name server resources and to create DNS records for any configured ingress/egress proxies

apiVersion: tailscale.com/v1alpha1
kind: DNSConfig
metadata:
 name ts-dns
spec:
  nameserver: {}

Wait for the DNSConfig to be ready and display nameserver’s Service IP:

$ kubectl get dnsconfig
NAME     NAMESERVERIP
ts-dns   10.44.14.137

Update Corefile to add a forward plugin for the nameserver IP:

ts.net {
    errors
    cache 30
    forward . 10.44.14.137
}

Create the desired tailscale Ingress with tailscale.com/experimental-forward-cluster-traffic-via-ingress label:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: eng
 labels:
   tailscale.com/experimental-forward-cluster-traffic-via-ingress: "true"
spec:
 rules:
 <backend specifications>
  tls:
 - hosts:
   - eng
 ingressClassName:tailscale

(See https://github.com/tailscale/tailscale/pull/11036 to understand why tailscale.com/experimental-forward-cluster-traffic-via-ingress label is needed)

Wait for tailscale to update Ingress status with the MagicDNS name of the ingress proxy

$ kubectl get ingress eng
NAME          CLASS       HOSTS   ADDRESS                      PORTS     AGE
eng     tailscale   *       eng.<tailnet-name>.ts.net   80, 443   81s

Cluster workloads as well as tailnet workloads should now be able to access the ingress backends over HTTPS (and over Wireguard in case of tailnet workloads) via eng..ts.net.

(Updated to remove a reference to a ProxyClass resource.

cc @flimamacedo

saadbahir commented 5 months ago

Currently this is being worked on in #11017 , #11019. Here is an approximate example user flow for accessing ingress proxy from within cluster that will be possible once the above linked work is done:

cc @flimamacedo

Hello @irbekrm Thank you for these details I recently posted another issue with a similar use case https://github.com/tailscale/tailscale/issues/11647

I want to be able to connect 2 kubernetes cluster through tailnet using MagicDNS.

Do you know whether this would be solved by your solution?

Thank you for your help

EDIT: If so, it seems that my issue is a duplicate of this one, correct?

irbekrm commented 5 months ago

I want to be able to connect 2 kubernetes cluster through tailnet using MagicDNS.

Replied in https://github.com/tailscale/tailscale/issues/11647#issuecomment-2043290328 I think it makes sense for that one to be a separate issue, we probably want to document that use case separately

irbekrm commented 4 months ago

The initial work for this has been merged, see https://github.com/tailscale/tailscale/issues/11500#issuecomment-2093375542 for instruction how to try this out. We are very keen to hear some feedback. Note that 1) this currently only works for Tailscale egress proxies and for proxies created for an Ingress resource 2) to be able to use this to refer to Ingress Tailscale MagicDNS name, you must also annotate the Ingress resource with tailscale.com/experimental-forward-cluster-traffic-via-ingress annotation (this is to make the proxy listen on its Pod IP, so that it can be reached by another Pod, not just over tailnet)

stennisrl commented 4 months ago

This is very exciting! Nice work @irbekrm

Are there any plans to allow for a pod to resolve arbitrary FQDNs on the Tailnet without the need for a matching egress service?

My primary use case is having a pod that deploys compiled binaries to machines on the Tailnet as part of a CI pipeline. I have a considerable number of machines (to the point where creating egress services for each one would be cumbersome), so it would be great to have this functionality built into the operator. Alternatively if there's a different way of accomplishing this use case, I'm all ears!

irbekrm commented 4 months ago

Hi @stennisrl at the moment this is aimed at operator managed mechanisms that expose tailnet services to cluster.

In generall I think allowing folks to populate the nameserver with arbitrary tailnet FQDNs would probably not be very useful since the cluster workloads would anyway not be able to reach the tailnet services on their tailnet IPs (in case of egress proxies, the records that we create are MagicDNS names -> proxy Pod IPs).

having a pod that deploys compiled binaries to machines on the Tailnet as part of a CI pipeline

I guess currently the best solution would be to deploy Tailscale in sidecar alongside this Pod. We have thought about making it easier to inject sidecars to cluster workloads for cases were proxies are not feasible (i.e via a mutating webhook), but currently there are not that many use cases for this. Have you tried using a sidecar for this? https://github.com/tailscale/tailscale/tree/main/docs/k8s#userspace-sidecar

irbekrm commented 4 months ago

This was released in 1.66. We are looking for more feedback for this feature, but I will close this particular issue as it's getting quite large- folks who would like additional functionality could open new issues.

tailscale / tailscale