Automatic HTTPS through Let's Encrypt

[Detry322]

(This is an enhancement)

It would be really cool if it were possible to automatically enable https through the new Let's Encrypt service, since it's now trusted by all major browsers. It's also part of the Linux Foundation so it's rooted in the Open Source community.

[titanous]

Yeah, we'd love to have this. Ideally they'd have support for wildcard certificates, but that's not a blocker.

[philiplb]

This would be totally awesome. No more worries about expired certificates, no more manual requesting them. They won't support wildcard afaik and the runtime is 90 days. They recommend to renew them actually like once per week? See https://letsencrypt.org/2015/11/09/why-90-days.html

[titanous]

Yeah, we'll have automated renewal set up.

[rikur]

+1 would save time and money when running multiple flynn environments (development, staging, production).

[temujin9]

:+1: - this would save me time, money, and frustration.

[christiangenco]

+1

[hadifarnoud]

+1 saves a lot of time.

[SystemZ]

Any news on this? Any tip how to renew cert without downtime before auto renew is implemented?

[titanous]

@SystemZ this is currently scheduled for work as soon as user authentication (currently in progress) lands.

There should be no issues renewing Let's Encrypt certificates without downtime, issue a new certificate (I like the this tool) and then use flynn route update to do a zero-downtime replacement of the certificate.

[SystemZ]

@titanous Thanks for fast info and tip with acme :)

[slava-vishnyakov]

After some digging here's a workaround using Caddy until Flynn has LetsEncrypt built-in:

BEWARE! This is a dirty hack... Use at your own risk.

Seriously! Dirty hack with some hardcoding ahead!

On your local computer (where flynn CLI is):

export DB=$(flynn -a controller env get PGDATABASE)

flynn -a postgres pg psql -- $DB <<EOF
SELECT processes->'app'->'args' 
  FROM releases 
  WHERE processes->'app'->'args'->>0 = '/bin/flynn-router';
EOF

This should show you something like this:

["/bin/flynn-router", "-http-port", "80", "-https-port", "443", "-tcp-range-start", "3000", "-tcp-range-end", "3500"]

If your output is different - modify the next command to match.

Replacing 80 with 82 and 443 with 445 - issue this command:

flynn -a postgres pg psql -- $DB <<EOF
UPDATE releases 
  SET processes = jsonb_set(
    processes, '{app,args}', 
    jsonb '["/bin/flynn-router", "-http-port", "82", "-https-port", "445", "-tcp-range-start", "3000", "-tcp-range-end", "3500"]') 
  WHERE processes->'app'->'args'->>0 = '/bin/flynn-router';
EOF

This will move flynn-router from ports 80 to 82, and from 443 to 445, so that Caddy can run (otherwise it can't request certificates).

Then run (again on local computer):

echo "
cat <<EOF | sudo tee /etc/ssl/private/flynn.crt
$(flynn -a postgres pg psql -- -qAt $DB -c "select env->>'TLSCERT' from releases where processes->'app'->'args'->>0 = '/bin/flynn-router';")
EOF
cat <<EOF | sudo tee /etc/ssl/private/flynn.key
$(flynn -a postgres pg psql -- -qAt $DB -c "select env->>'TLSKEY' from releases where processes->'app'->'args'->>0 = '/bin/flynn-router';")
EOF
chmod 0600 /etc/ssl/private/flynn.*
"

This will output for you a set of instructions - you need to run them on flynn-host computer (i.e. where cluster is) -- this will export your "controller" certificates for Caddy (otherwise your flynn CLI will stop working, complaining about certificate pinning).

Warning I have no idea when those expire and how they are managed, if you lose access to your controller - change your .flynnrc URLs (2 of them) to https://...:445 (That is also an alternative to exporting certificates, but the dashboard won't work, since it relies on port 443 heavily. And Caddy can't request ceritificates for *.flynnhub.com for some reason..)

After you run the above commands, run those on flynn-host system:

wget -qO- https://getcaddy.com | bash

Beware! bash|curl is lazy and insecure. It's better to install by hand! https://caddyserver.com/download

Then run those on flynn-host replacing your e-mail with real one (for Let's Encrypt)

export MY_EMAIL=mail@mail.com

cat <<EOT > /etc/init/caddy.conf
#!upstart

description "caddy"

start on runlevel [2345]
stop on runlevel [!2345]

console output

env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

exec /usr/local/bin/caddy --conf /etc/caddy/Caddyfile --agree
respawn
EOT

cat <<EOT > /etc/caddy/Caddyfile
https://* {
  tls /etc/ssl/private/flynn.crt /etc/ssl/private/flynn.key
  proxy / https://127.0.0.1:445 {
    insecure_skip_verify
    header_upstream Connection {>Connection}
    header_upstream Upgrade {>Upgrade}
    header_upstream Host {host}
    header_upstream X-Real-IP {remote}
    header_upstream X-Forwarded-For {remote}
    header_upstream X-Forwarded-Proto {scheme}
  }
}

example.domain.com othersite.com {
  tls $MY_EMAIL
  proxy / http://127.0.0.1:82 {
    header_upstream Connection {>Connection}
    header_upstream Upgrade {>Upgrade}
    header_upstream Host {host}
    header_upstream X-Real-IP {remote}
    header_upstream X-Forwarded-For {remote}
    header_upstream X-Forwarded-Proto {scheme}
  }
}
EOT

stop flynn-host
start flynn-host
start caddy

(Note: in Caddy 0.9.0 with this configuration the dashboard doesn't quite work. you need to download the latest git and build it to make dashboard (and websockets) work)

In latest (git master) Caddy this is enough (in 0.9.0 this doesn't work):

https://* {
  tls /etc/ssl/private/flynn.crt /etc/ssl/private/flynn.key
  proxy / http://127.0.0.1:82 {
    websocket
    transparent
  }
}

Warning This will run Caddy as root, which is not recommended. Alternatively you can wget -qO /etc/init/caddy.conf https://raw.githubusercontent.com/mholt/caddy/master/dist/init/linux-upstart/caddy.conf, but this runs Caddy as www-data and that user will need to have access to your certificates. I don't know which would be better.

You will need to change example.domain.com (othersite.com is added just as example) to your hosted site in /etc/caddy/Caddyfile and restart caddy

(I'm not affiliated with Flynn team, so again - use at your own risk, this recipe might be incredibly stupid for all I know :) )

_To revert: _

on Flynn host:

stop caddy
rm /etc/init/caddy.conf

on local machine:

export DB=$(flynn -a controller env get PGDATABASE)
flynn -a postgres pg psql -- $DB <<EOF
UPDATE releases 
  SET processes = jsonb_set(
    processes, '{app,args}', 
    jsonb '["/bin/flynn-router", "-http-port", "80", "-https-port", "443", "-tcp-range-start", "3000", "-tcp-range-end", "3500"]') 
  WHERE processes->'app'->'args'->>0 = '/bin/flynn-router';
EOF

on Flynn host:

stop flynn-host
start flynn-host

[benosman]

I've seen the following golang library vendored a few times in other software to assist with LetsEncrypt / ACME integration. Supports HTTP and DNS validation.

https://github.com/xenolf/lego

Just putting it here in case you guys haven't come across it as yet.

[titanous]

@benosman Thanks! We're aware of the packages available and plan to use this one (I've sent a few patches already): https://godoc.org/golang.org/x/crypto/acme

[ghost]

Is this dirty hack the only option to get let's encrypt support manually?

[titanous]

@dropfen no, you can use the -manual flag with DNS or HTTP (hosted on Flynn) from your local machine with this tool: https://github.com/google/acme

[ghost]

It sounds good, but I don't have a clue how to do it really :)

[ghost]

If someone knows what @titanous mean, please take some minutes if you can, to describe how to get letsencrypt work with flynn. Would be really helpful for a lot of people. thank you guys

[ghost]

@titanous Ok, it was really easy, actually, thank you for the tip 👍

[Banashek]

@dropfen care to post a small description? I'm just getting started with flynn (been using straight containers with compose) and would love to try out integrating letsencrypt.

[ghost]

Sure, no problem:

• follow the instructions here: https://github.com/google/acme (you should end up with cert & key in ~/.config/acme/)
• then update the route for the domain

flynn route <route-id> update --tls-key ~/.config/acme/your-domain.key --tls-cert ~/.config/acme/your-domain.crt

hint: to get the route id, just call flynn route

[mbreedlove]

@dropfen Won't this be a problem when you have to renew the certificate? It looks like you would have to manually stop the application to renew the certificate via Acme.

I've been using the Dokku Let's Encrypt plugin for a while, it just adds a new nginx location block to intercept the Let's Encrypt domain verification URL so the application can continue running, uninterrupted.

@titanous, is there a way to let the Acme client perform verification while the Flynn application is running?

[titanous]

@mbreedlove A few points:

1. The authorizations are currently valid for 10 months, so it's likely that the renewal will not require a new authorization and will be issued immediately (this has been my experience).
2. Path-based routing can be used to send the requests to a different app.
3. An endpoint can be added to the app for the LE HTTP authorization.
4. DNS authorization can be used.

None of these options require any downtime or interruption.

[ghost]

@mbreedlove to get a new cert you can use acme cert -manual example.com Then, you would have to put challenge file in the .well-known/acme-challenge dir and letsencrypt would just call it, to verify your host. And finally with flynn route update you can put the new cert again into the router. So no downtime is required.

Note: you don't have to use a file, for me (phoenix framework) it didn't worked with .dot_folders like .well-known, but it worked with a route.. So, it would be also possible to keep your challenges in the DB and manage them over the UI, for example..

[philiplb]

What just came into my mind: Although I will totally use this, there might be people who won't. So this should be maybe opt-in? Also because it would let Flynn depend on a third party service which might go down the next day.

[titanous]

@philiplb It will be opt-out. The native security win is too compelling to require people to work to take advantage of it. I'm confident that they won't disappear tomorrow and if there are any issues we'll evaluate carefully.

[hadifarnoud]

is this going to be per route or per app? we're using one app with so many domains, per app would make it hard to use

[titanous]

@hadifarnoud Yes, per route.

[kations]

@titanous could you assume how long it takes to use this feature in prod?

[thomasmodeneis]

@dropfen the correct usage for adding the certificate has changed, now is (if you are in the same folder as your app is) flynn route <route-id> update -k ~/.config/acme/your-domain.key -c ~/.config/acme/your-domain.crt

Or from any folder, using -a

flynn -a yourapp route <route-id> update -c ~/.config/acme/your-domain.crt -k ~/.config/acme/your-domain.key

[txchen]

This feature will be super useful for those who don't have a wildcard cert.

[michael-bouvy]

Very useful thread, thanks. Working command is:

flynn route update <route-id> -k <path-to-key>.pem -c <path-to-fullchain>.pem

[hadifarnoud]

@michael-bouvy how do I get letsencrypt to verify my domain? that's needed in order to get pem files

[michael-bouvy]

@hadifarnoud take a look at https://github.com/lukas2511/dehydrated

[michael-bouvy]

Indeed, I did use DNS challenge to get this working.

[hadifarnoud]

it works for some subdomains and don't for others even though I set all subdomains correctly. how can I reverse the route update?

[michael-bouvy]

flynn route remove <route-id>

[puterleat]

Is anyone actually working on LE integration right now? I'm weighing up the effort of hacking something together vs waiting and/or helping do it right. (EDIT - I cant see a LE branch, for example)

[nemanjan00]

Guys. This is the only reason I am using Dokku and not Flynn. :D

How can I help?

Please, can we speed this up?

Thank you!

[nemanjan00]

Dokku does this automatically, with one command.

Setting it all up is too slow.

It already takes my more than 2h to bootstrap each project I start...

[andyjeffries]

+1 We're using Dokku in production for our app and API (along with a number of personal projects) and I'd love not to be. The two reasons for that is that when I tried an older version of Flynn it had a tendency to grow disk space usage over time and never reclaim it and Lets Encrypt support. I'd love to help this feature that's been open for over a year (when both Caddy and Dokku support it out of the box).

[DoubleMalt]

Is there anyone working on it?

[clementvp]

Is there anyone working on it?

Dokku use a let's encrypt plug-in, and is so easy to get in, but doesn't support clustering. Flynn is production ready and support clustering but doesn’t support let's encrypt.

It's a PaaS hard choice ;)

Wildcard certificate are SO expensive for little start up :/

[OmgImAlexis]

@clementvp let's encrypt is adding wildcard certs.

[clementvp]

@OmgImAlexis Not before january 2018 right ?

[girvo]

I'm also curious as to what the progress is like for this, or at least where I should start to look to add it myself! It's the only missing feature as far as what I require, so it'd be a massive boon to be able to get it in the core 😀

[titanous]

The current plan is to target ACME v2/wildcards, which will be landing in January. It's likely that implementation pieces will land before that.

[llim5432]

Any updates on this?

[roquie]

I also waiting for updates…

[shalakhin]

Me too

[lmars]

Work is in progress, I pushed a proof of concept last week (see this commit) which is functional but untested.

If you want to try it out in a test / staging environment then install version v20180105.dev1 and set LETS_ENCRYPT_KEY to a PEM encoded private EC key during bootstrap, for example:

export LETS_ENCRYPT_KEY="$(openssl ecparam -genkey -name prime256v1 -noout)"
export CLUSTER_DOMAIN="my-domain.com"
flynn-host bootstrap