💡Automatically update DNS records when new service is added

[scubbo]

Describe the feature you'd like When adding a new service (available locally on service.internal) to Cloudflared tunnel (with UUID <UUID>) so that foo.example.com resolves to that service, two steps are required:

• Update the local Cloudflared daemon's config, mapping hostname: foo.example.com to service: service.internal
• Publish a CNAME record so that foo.example.com maps to <UUID>.cfargotunnel.com

The first step is naturally achieved by updating the configuration and restarting the daemon. However, I couldn't find a way to automate the second step. In particular, the cloudflare/cloudflared image does not appear to contain any tools (like yq, awk, or Python) which could be used to parse the config file to generate the appropriate cloudflared tunnel route dns <tunnel_name> <domain_name> calls per-service.

This feature would be either:

• A flag on cloudflared tunnel run (like --update-dns) which results in updates to the DNS records (if hosted on Cloudflare) before the Tunnel starts.
• An execution mode for cloudflared tunnel route dns <tunnel_name> which results in per-service calls for all services defined for that tunnel.

Describe alternatives you've considered As described in this blog post, I've created a Docker image for myself which can do the "parse Cloudflared config file, make cloudflared tunnel route dns <tunnel_name> <domain_name> calls" logic itself as an initContainer on a Kubernetes deployment. If and when I publish this image to Docker Hub, other users could use it in their own setups.

Additional context I recognize that this feature would result in some extra traffic to Cloudflare's DNS services, since every deployment of a Tunnel would result in updates for every service it fronts. Some of this traffic could be mitigated by implementing a check to only make DNS-update calls for records that don't currently exist. I would guess, though, that this extra traffic (which would only be incurred for users who actually enable this feature) would be inconsequential against Cloudflare's overall traffic, since tunnel re-deployment rate is probably(?) pretty low.

Given that this feature wouldn't involve much new logic and would just be plumbing existing logic into a new command flag, I'd be happy to take a stab at implementing this feature if it's considered helpful. I've done some Go tutorials but have never used it professionally.

[fernvenue]

Hi @scubbo, I'm not the Cloudflare employee, but I think you can try using Zero Trust Dashboard to do that, just follow the documentation. I now use Dashboard entirely to configure services on cloudflared, including adding, deleting, and modifying, on Dashboard you will not need to manually configure DNS records.

But yeah, automatically update DNS records also helpful when we set up tunnel locally, I agree with that :)

[scubbo]

Thanks @fernvenue! This wouldn't suit my needs as I'm looking for an automatable solution (ideally, one that is driven by configuration which can be reviewed, stored in Source Control, deployed with CI/CD, etc.) - but it's always good to know of alternative approaches!

[DevinCarr]

There are some great points here around usability when creating a tunnel for the first time from the command-line. We have purposefully opted to not combine the operations of allocating a DNS entry for a tunnel during the creation of a tunnel. Originally, this used to be the case, but it was the combination of two different operations: tunnel creation and DNS entry creation fused together. This ended up causes issues during a failure of either and reporting to the user which failed and how to manage it.

So we split the creation of a tunnel separate from the creation of a DNS record to point to the tunnel. Additionally, there are some tunnels that do not need to be exposed to the public internet via the DNS record and are setup as route via the ZT private network.

Typically the design and deployment we encourage is to create your tunnels up-front and allocate them into areas or services that need to use them via the cloudflared tunnel run --token <token>. This allows you to manage the tunnels offline from your current system and don't need to worry about tunnel creation errors during the deployment of your services. You will just be limited to errors that might occur to run the tunnel.

Feel free to let me know if this helps you understand a bit more and post more questions otherwise!

[scubbo]

Thanks for the context, Devin! It's always helpful to understand the history and motivation of a tool's design. However, I think we're talking about two different use-cases - it's possible that I haven't explained mine well enough, or perhaps I'm using the tool in a discouraged or unforeseen way. I'm not talking about creating a tunnel for the first time, but rather about updating the configuration of an existing tunnel to add a new hostname->service mapping. Let me try to clarify.

In my current setup, I have a single Tunnel (i.e. a single entry, with a single UUID, shows up on the ZeroTrust dashboard under "Access -> Tunnels"). This abstract Tunnel is concretely implemented by two daemons (with identical configuration. Two daemons allow ""updat[ing] the configuration of a tunnel without downtime") that run on a Kubernetes cluster. These daemons are configured to provide routing to all appropriate services that are either hosted on the Kubernetes cluster, or that are otherwise available on my private network. Please excuse the crude diagramming:

(The internal DNS lookups within the Kubernetes cluster have been omitted from the sequence diagram since they're irrelevant)

(I don't think the fact that I'm running Cloudflared on Kubernetes is relevant - I believe this same use-case would still be relevant if it was being run as a standard Systemd service, or even hand-managed by manually running cloudflared tunnel run <args> - but I'm including that information just in case it sheds further light on how my use-case differs from your expectations)

So far, so standard - I doubt that this is unusual. However, the difference between my process and what you described is that, when I add a new origin, I'm not creating a new tunnel - rather, I'm just adding a new mapping to the daemon config. To be painfully precise, the process is:

1. Start up (either via Kubernetes, or otherwise) a service that will be a new origin - confirm via private-network-internal calls that it is appropriately operational.
2. Update the Cloudflared config.yaml to include a mapping from the desired external hostname, to the private-network-internally-available address.
3. Deploy the changed config.yaml to all Cloudflared daemons (in my setup, this is by updating a Kubernetes deployment which will kill existing pods and bring up new pods that will consume the new configuration; in other setups, this could be by killing and restarting the daemons manually). Note that at this point the origin is still not available externally on the desired hostname, since we also have to...
4. (This is the step I'm hoping to automate!) ...add a DNS record for the desired hostname (alongside the existing DNS records for previously-configured hostnames) to be a CNAME pointing to <UUID>.cfargotunnel.com - by calling cloudflared tunnel route dns <tunnel_name> <hostname>, or by using the Cloudflare Dashboard to create a DNS entry, or by using the Cloudflare API directly.

My request is for a way for steps 3 and 4 to be merged - that is, when a new daemon for an existing tunnel is brought up, it will (optionally! As you say, not all tunnels are intended to be publicly-accessible) add any missing CNAME records (by effectively calling cloudflared tunnel route dns <tunnel_name> <hostname> for all defined hostnames, or by using explicit Cloudflare APIs; though that latter would require a different form of authentication).

It's worth noting that there's an existing guide on how to serve multiple origins with a single Tunnel, so I'd be surprised to hear that adding a new origin to an existing tunnel is an unsupported use-case.

[DevinCarr]

Alrighty, so first off, great diagram!

Secondly, we typically operate cloudflared operations from a static origin point-of-view. We don't expect the origins to be dynamically added or removed services behind cloudflared (outside of replicas: multiple hosts of the same origin). With that said, you still should be able to combine steps 3 and 4 as you described by leveraging the Cloudflare API. More specifically:

• Update a configuration to support an additional origin (or to add a few): https://api.cloudflare.com/#cloudflare-tunnel-configuration-put-configuration I'll admit, that API isn't the best for what you are attempting by patching the current, but you should be able to GET the current config and PUT in the new origins.
• Then adding a new DNS record for the origin: https://api.cloudflare.com/#dns-records-for-a-zone-create-dns-record

After this setup, your cloudflared will get pushed the latest configuration and then the Cloudflare Edge will receive the new DNS record and traffic should begin to flow to your newly minted origins.

The main reason we don't combine these operations is the same reason there are two API operations to do each step. We try to limit our cloudflared commands to one API operation each where possible. Additionally, all the API calls made in cloudflared are the same API calls that you can make with the public Cloudflare API directly.

I agree that we might be able to help around some of the ergonomics here for adding new origins with more ease (maybe by providing a PATCH to the current config with new origins) and I'll ping @abelinkinbio to see if he can review this further.

[scubbo]

We don't expect the origins to be dynamically added or removed services behind cloudflared (outside of replicas: multiple hosts of the same origin)

Got it. That totally explains why this (to me!) "obvious" papercut exists - because I'm trying to do something (adding new origins) that isn't anticipated. Makes sense - thanks!

I'm intrigued to learn that it's possible to update a cloudflared daemon's configuration by making an API call. I had been under the impression that the only way to update a daemon's configuration was by restarting it with new (locally-defined) configuration, but it looks like you're saying that it's possible to update the (abstract, cloud-hosted) tunnel definition and that will get deployed to the daemons. That's neat, thanks - I'll play around with that!

Happy to see what abe thinks. I'd also be content to conclude with "you shouldn't be dynamically adding new origins [implicitly - you should create a new tunnel for each origin or set-of-replicas-of-origins], so this feature shouldn't need to exist", that seems like a reasonable position.

[christidis]

This issue is still Open so I will put a reply anyway. For 4 for the DNS part I am leveraging external-dns for managing my CNAME records for my tunnelled applications dynamically. This works pretty well for me.

Also for 3 since cloudflared does not support dynamic config map reloads you may be able to switch to a Zero Trust Managed Tunnel (so cloudflared just pulls the configuration dynamically) and then bootstrap Zero Trust "Applications" in Cloudflare along with their origins configuration. For this you will need a tool (like external-dns or write your own) but for the origin CRDs..

[scubbo]

I'd quite forgotten about this issue - thanks for the reminder!

You can see my solution here - a simple shell script that uses the cloudflared CLI to update DNS, and which can be used as an initContainers on deployment of a cloudflared pod.

[christidis]

404s / private repository

[scubbo]

Whoops - thanks for letting me know, repo should now be public.

[aur3l14no]

@scubbo Hey I just googled here. Repo is private now. Can you set it public again? Thanks.

[scubbo]

@aur3l14no Done, sorry about that - I always forget that, when I reinstall Gitea, repos default to private.

[nickchomey]

@scubbo would you mind making it public again?

[scubbo]

Done 🙃

[nickchomey]

@scubbo thanks! Though I did some more digging after my previous comment and found that using cloudflared can't work for my needs.

This is because when you do locally managed (cli) tunnels, you first login by selecting a specific domain/zone. If you do cloudflared tunnel route dns, it can only add cnames for that originally selected domain/zone. I'm sure that's fine for many, but I want to have different zones managed/routed by the same tunnel.

Conversely, with a dashboard managed tunnel, it can make routes to any zone in your account.

My solution is to either use a direct http api call to set the CNAME (as was suggested in this thread), or use the flarectl utility which can do api operations quite easily (note that you specify the zone name/domain rather than the zone id when setting dns with it)

More details in this support convo in discord, which confirms how cloudflared is a relatively neglected tool... https://discord.com/channels/595317990191398933/1253128218890997810