Feature: Add resources for Argo Tunnel

[abeluck]

Currently when using argo tunnel one must manually login with cloudflared login on each server you want to tunnel with Argo. This prevents using Argo tunnel at scale.

The cert.pem file created by cloudflared login consists of the CA Origin Cert, Private Key, and an ARGO TUNNEL TOKEN.

The existing resource origin_ca_certificate takes care of the origin cert and private key, but we need an additional resource to provision argo tunnel tokens.

Terraform Version

terraform v0.12.20

[jacobbednarz]

Thanks for raising this @abeluck! I'm unable to find any API documentation on this; are you able to point it out to me and we can chat about the next steps?

[abeluck]

many thanks for the quick reply. Yea, so I'm not sure there is a documented API for creating the token.

The cert.pem format cloudflared expects looks like this:

-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN ARGO TUNNEL TOKEN-----
...
-----END ARGO TUNNEL TOKEN-----

It is described in this FAQ:

Cloudflare generates a certificate that consists of three components:

1. The public key of the origin certificate for that hostname
2. The private key of the origin certificate for that domain
3. A token that is unique to Argo Tunnel

Those three components are bundled into a single PEM file that is downloaded one time during that login flow. The host certificate is valid for the root domain and any subdomain one-level deep. Cloudflare uses that certificate file to authenticate cloudflared to create DNS records for your domain in Cloudflare.

The third componenent, the token, consists of the zone ID (for the selected domain) and an API token scoped to the user who first authenticated with the login command. When user permissions change (if that user is removed from the account or becomes an admin of another account, for example), Cloudflare rolls the user’s API key. However, the certificate file downloaded through cloudflared retains the older API key and can cause authentication failures. The user will need to login once more through cloudflared to regenerate the certificate. Alternatively, the administrator can create a dedicated service user to authenticate.

I have verified this by taking the contents between BEGIN/END ARGO TUNNEL TOKEN and base 64 decoding it. It consists of two lines: a zone id, and an api token:

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
v1.0-1234zbbcd....

The zone id is simple enough to provide. It is the token that needs to be generated with terraform.

According to the api docs, API tokens that begin with v1.0- are "User Service Key". In the Cloudflare UI, your "Origin CA Key" has the same format. However the token in the argo cert.pem is not the same as my Origin CA Key, which means they are creating a new one.

And this is where my investigation hits a brick wall. I can't find any API docs for provisioning User Service Keys.

I suppose Cloudflare will have to step in and help us to move this along.

[jacobbednarz]

Have you raised a support case? If not, are you able to submit one and see if the API can be publicly documented we can chat about adding support for it?

[abeluck]

Good idea!

I'll open a ticket with CF support and get back to you here if anything comes of it.

Is CF involved in the development of this provider, is it a community project or is HashiCorp responsible? (just curious!)

[jacobbednarz]

Is CF involved in the development of this provider, is it a community project or is HashiCorp responsible? (just curious!)

Both :) One of the other primary maintainers is a Cloudflare employee and others drop in occasionally to help out with their products.

[abeluck]

An update here, over on the Cloudflare forum a cloudflare engineer shared this script:

#!/bin/bash

## example parameters, adjust as needed
ORIGIN_CA_KEY="${CLOUDFLARE_ORIGIN_CA_KEY}"
CF_API_KEY="${CLOUDFLARE_AUTH_KEY}"
CF_EMAIL="${CLOUDFLARE_AUTH_EMAIL}"
TUNNEL_ZONE_ID="${CLOUDFLARE_ZONE_ID}"
TUNNEL_HOSTNAMES=""
case $STAGE in
  development)
    TUNNEL_HOSTNAMES='["db-dev.exmaple.com"]'
    ;;
  production)
    TUNNEL_HOSTNAMES='["db.exmaple.com"]'
    ;;
  *)
    exit 1
    ;;
esac

## now for the actual script:
set -e

## tmp dir
tmpbase=/tmp/${0##*/}
find ${tmpbase}* -type f -exec shred -uvxz {} ';' || true
rm -rf ${tmpbase}*
TMPDIR="`mktemp -d ${tmpbase}-XXXXXX`"

curl -s https://api.cloudflare.com/client/v4/user/service_keys/origintunnel \
  -H "x-auth-key: $CF_API_KEY" \
  -H "x-auth-email: $CF_EMAIL" \
  | jq -r .result.service_key \
  > $TMPDIR/tunnel_service_key.txt

# generate private key
openssl ecparam -name prime256v1 -out $TMPDIR/tunnel_private_key_params.txt
openssl req -batch -new \
        -newkey ec:$TMPDIR/tunnel_private_key_params.txt \
        -nodes -out ${TMPDIR}/csr.txt \
        -keyout ${TMPDIR}/tunnel_private_key.txt \
        -subj "/C=US/CN=CloudFlare"

# make cert.pem, containing
# 1. Private key (in PKCS #8 format)
openssl pkcs8 -topk8 \
        -in ${TMPDIR}/tunnel_private_key.txt \
        -nocrypt -out ${TMPDIR}/cert.pem

# 2. public key from originCA
curl -s -XPOST https://api.cloudflare.com/client/v4/certificates \
  -H "Content-Type: application/json" \
  -H "X-Auth-User-Service-Key: $ORIGIN_CA_KEY" \
  -d "$(jq -n --arg csr "$(cat ${TMPDIR}/csr.txt)" --argjson hostnames "$TUNNEL_HOSTNAMES" '{hostnames:$hostnames,requested_validity:5475,request_type:"origin-ecc",csr:$csr}')" \
  | jq -r .result.certificate \
  >> ${TMPDIR}/cert.pem

# 3. Argo Tunnel token
echo "-----BEGIN ARGO TUNNEL TOKEN-----" >> cert.pem
echo -n "$(echo $TUNNEL_ZONE_ID; cat ${TMPDIR}/tunnel_service_key.txt)" | \
        base64 | fold -w 64 >> cert.pem
echo "-----END ARGO TUNNEL TOKEN-----" >> cert.pem

if [ "x$DEBUG" == "x" ]; then
  find ${tmpbase}* -type f -exec shred -uvxz {} ';' || true
fi

This can be used to generate the cert.pem file cloudflared requires!

It uses the endpoints:

• GET /client/v4/user/service_keys/origintunnel
• POST /client/v4/certificates

@jacobbednarz Is this sufficient information to create a terraform data resource?

[jacobbednarz]

I can't find any public references to /user/service_keys/origintunnel so we'd probably want to get that documented somewhere first.

Once that's ready, we can add something to cloudflare/cloudflare-go to hide the complexities of this and then call that method somewhere in Terraform for use.

[abeluck]

so we'd probably want to get that documented somewhere first.

@jacobbednarz Any idea who to ping about this? My support request was closed with a vague, "ok, thank you" response.

The script itself works well enough, but it really belongs in terraform-provider-cloudflare.

[jacobbednarz]

Support is the best place. They are quite good at routing requests to the engineering team and PMs if needed. You could also ask for a rough timeline or to be notified when the functionality has been documented. I’m not sure what that is like with a non-enterprise account but it works pretty well with enterprise ones.

You’re free to add the functionality to your own forked version of the Terraform Provider if you really want it now. However, I’m not going to review or entertain the idea of merging it until there is a stable endpoint.

[bennesp]

Since this is an old (but very precious) issue, I would like to update future people (or the future me):

In the meanwhile, the format of the argo tunnel token has changed: now it is the base64 of a JSON like this

{"zoneId":"123...","accountId":"456...","serviceKey":"v1.0-789...","apiToken":"abc..."}'

Actually, accountId and apiToken don't seem to be used in my case, but I may be using argo tunnel in some wrong way.

If you need to get an apiToken, I suggest you perform a

cloudflared tunnel login

which will open the browser, and then you can inspect the calls in the Network tab when clicking on "Authorize".

[trjstewart]

We've recently run into a use for Argo and would like to make some effort towards getting this supported. Thankfully we're an Enterprise customer with a bit of swing so I'll get a support ticket created and see what I can get done about solidifying the API and getting some public documentation.

Support ID 2009906

[jacobbednarz]

Awesome @trjstewart! I'll poke some engineering folks and our account team as well to see if we can get some eyes on this too.

[bennesp]

Don't know if this can help, but here's the script we are using.

Like the previously posted, but slightly modified to fit the "new" argo tunnel token

#!/bin/sh

# ----- BEGIN CONFIGURATION -----
# Used to exchange CSR for Certificate
ORIGIN_CA_KEY="v1.0-123456..."
TUNNEL_HOSTNAMES="[\"www.abc.def\",\"abc.def\"]"
VALIDITY_DAYS=5475

# Used to get an argo tunnel token
CF_API_KEY="abcdef..."
CF_EMAIL="email@email.email"
ARGO_ZONE_ID="ghijkl..."
# ----- END CONFIGURATION -----

# Uncomment if in a CI, to help debugging
# set -x
set -e

TMPDIR=$(mktemp -d)
trap "rm -rf $TMPDIR" EXIT INT

# Generate private key
openssl ecparam -name prime256v1 -genkey -out "$TMPDIR/private_key.pem"

# Generate CSR
openssl req -batch -new -key "$TMPDIR/private_key.pem" -out "$TMPDIR/csr.pem" -subj "/C=US/CN=CloudFlare"

# Private key pkcs1 to pkcs8
openssl pkcs8 -topk8 -in "${TMPDIR}/private_key.pem" -out "${TMPDIR}/private_key_pkcs8.pem" -nocrypt

# Request to cloudflare to exchange the CSR with a Certificate
curl -s -XPOST https://api.cloudflare.com/client/v4/certificates \
  -H "Content-Type: application/json" \
  -H "X-Auth-User-Service-Key: $ORIGIN_CA_KEY" \
  -d "$(jq -n --arg csr "$(cat ${TMPDIR}/csr.pem)" --argjson hostnames "$TUNNEL_HOSTNAMES" '{hostnames:$hostnames,requested_validity:$VALIDITY_DAYS,request_type:"origin-ecc",csr:$csr}')" \
  | jq -r .result.certificate > ${TMPDIR}/cert.pem

serviceKey=$(curl -s -H "x-auth-key: $CF_API_KEY" -H "x-auth-email: $CF_EMAIL" https://api.cloudflare.com/client/v4/user/service_keys/origintunnel | jq -r .result.service_key)

# Now we have all the information needed to forge an Argo Tunnel Certificate

# accountId and apiToken seem to be useless when connecting to argo
accountId=""
apiToken=""
argoCertContent="$(jq -n \
    --arg zoneId "$ARGO_ZONE_ID" \
    --arg accountId "$accountId" \
    --arg serviceKey "$serviceKey" \
    --arg apiToken "$apiToken" \
    '{"zoneId":$zoneId,"accountId":$accountId,"serviceKey":$serviceKey,"apiToken":$apiToken}')"
argoToken="$( echo "-----BEGIN ARGO TUNNEL TOKEN-----"; (echo $argoCertContent) | base64 | fold -w 64; echo -----END ARGO TUNNEL TOKEN-----)"

argoCert="$(cat "${TMPDIR}/private_key_pkcs8.pem")
$(cat "$TMPDIR/cert.pem")
$argoToken"

echo "$argoCert"

[jacobbednarz]

Appreciate it @bennesp but the blocker at the moment is lack of publicly documented endpoints for the functionality in use. As a rule of thumb, we shouldn't be building end user facing reliance on undocumented functionality as we don't have confirmation (in the form of documentation) that the endpoint is intended for use or intending to stay around. It is fine if you're in a jam to take whatever API endpoints you can find but it doesn't provide any guarantees it will be there tomorrow and we don't want to make a habit of introducing tooling or functionality that people cannot rely on. It leads to a poor experience for the maintainers and users of the functionality.

[trjstewart]

To add to the point, the API has already changed once. As much as I want the functionality, I also want it to be stable. I agree that waiting on docs or a confirmation that it's not going to change at the very least is necessary. With any luck two Enterprise customers asking about it will drive it home.

[bennesp]

I understand, and it makes perfect sense to me 😄 Hoping the best

[analytically]

Also something we're looking into given the Argo tunnel change making it harder to automate setup. https://blog.cloudflare.com/argo-tunnels-that-live-forever/

[abeluck]

As of a few weeks ago the old token style still worked. I believe that is still the case.

However, I definitely agree we want a stable API to build on.

Thanks for linking that new info @analytically .

[abeluck]

Alright. Recently we have some movement here. This is now possible!

There now exists a public API for CRUDing, Argo Tunnels: https://api.cloudflare.com/#argo-tunnel-create-argo-tunnel

With this API, and the new tunnel changes linked by @analytically we now have what we need to create a tunnel and tunnel credentials.

I've created a post on the cloudflare forums that show how to create and provision tunnels programatically.

For those curious about all the previous discussion of tokens: In effect there are no more argo tunnel tokens (at least in this new architecture). Rather a tunnel is now a named resource that you can manage via the API. Every tunnel is created with a secret value. Any cloudflared can run the tunnel given just a few params (the shared secret, the tunnel id, and account id).

As far as the terraform cloudflare provider is concerned, we need terraform resources for the argo tunnel apis. These are now public. I've updated the ticket title to reflect this new situation.

[abeluck]

Here is an example of how I expect argo tunnel creation to work, once the cloudflare_argo_tunnel resource has landed:

provider "cloudflare" {
  version              = "~> 2.0"
}

variable "tunnel_name" {
  type        = string
  description = "the name of the argo tunnel"
}
variable "tunnel_secret" {
  type        = string
  description = "a long random string that gives the possesor the ability to run the tunnel"
}
variable "tunnel_subdomain" {
  type        = string
  description = "the subdomain (not including the apex domain) that the tunnel will be served on"
}
variable "cloudflare_zone_id" {
  type        = string
  description = "the zone id the subdomain exists in"
}
variable "cloudflare_account_id" {
  type        = string
  description = "the account id all this is happening in"
}

# this resource doesn't exist yet
resource "cloudflare_argo_tunnel" "example" {
  secret = var.tunnel_secret
  name   = var.tunnel_name
}

resource "cloudflare_record" "example" {
  zone_id = var.cloudflare_zone_id
  name    = var.tunnel_subdomain
  value   = "${cloudflare_argo_tunnel.example.id}.cfargotunnel.com"
  type    = "CNAME"
  proxied = true
}

resource "local_file" "credentials" {
  content = jsonencode({
    "AccountTag"   = var.cloudflare_account_id,
    "TunnelSecret" = var.tunnel_secret,
    "TunnelID"     = cloudflare_argo_tunnel.example.id,
    "TunnelName"   = var.tunnel_name
  })
  filename = "${path.module}/${var.tunnel_id}.json"
}

resource "local_file" "config" {
  content = yamlencode({
    "url"    = var.origin_url,
    "tunnel" = var.tunnel_name
  })
  filename = "${path.module}/${var.tunnel_id}.yml"
}

[jacobbednarz]

I'll have to have a look at this when I'm in front of the code next but I think we're still pending cloudflare-go support for this as well.

[jacobbednarz]

PR adding support to cloudflare-go has been opened at cloudflare/cloudflare-go#567. Once that lands, we can incorporate it as a resource here.

[jacobbednarz]

the changes in cloudflare-go are now in this repository and ready for implementation

[jacobbednarz]

905 is up for review. it does depend on cloudflare/cloudflare-go#572 however the functionality is all the same on this end.