Ability to cron renewals without --daemon

[emmetog]

What would you like to be added

The current way of renewing requires starting a daemon which is difficult to manage using automated provisioning tooling. It could be simplified if there were a command that could be added to a crontab and run regularly (eg every minute, every hour, according to your tolerance). This command would check if a cert needs to be renewed (using the already existing ----renew-period flag presumably) and rotate certs only when that threshold is passed.

An example of the command that could be added to a crontab, which would only renew if certs expire within 4 hours from now, and do nothing otherwise:

step ca renew <path-to-crt> <path-to-key> --renew-period 4h

An alternative solution is to make a tighter integration with systemd and family to make the renew daemon easier to run, but this may add unneeded complexity, I'd suggest that a cronable command removes all that "daemon" complexity and gives more flexibility to the end user.

Why this is needed

Currently the suggested way of renewing certs is by using the command:

step ca renew <path-to-crt> <path-to-key> --daemon

This hangs, so to make it run permanently, you'd need to nohup it. This is fine for small deployments where the daemon is run manually but at scale or using automated provisioning tooling it's complicated to check if the renew daemons are running, you'd need to check if the process is running, starting if not. If multiple certs are managed then you'd need to check each one. Doing this idempotently is cumbersome.

[emmetog]

Workaround is to use --force and cron the command to renew every X hours/minutes, but a larger time between renewals means that there is more time for things to go wrong in the event of network outages. Smaller intervals fix this but at a cost of renewing certs far more than necessary.

[mmalone]

Hey @emmetog, thanks for the feedback.

Good news: I think we have everything you need. I'm going to leave this ticket open though since we clearly need to add this to documentation.

I hope I've addressed all of your concerns. If not, please let me know!

First, the step ca renew subcommand will prompt you interactively to overwrite existing files. If you're running step ca renew via cron or some other automation, pass the --force flag to avoid this prompt.

cron-based renewal

If you're using the "one-shot" renewer (without the --daemon flag) you can request that certificate renewal only occur if the certificate is approaching its expiry using the --expires-in <duration> flag. The <duration> is a time interval like "4h" or "30m". Renewal will only occur if the expiry is within <duration> of the current time. For example:

$ step ca renew --force --expires-in 4h foo.crt foo.key
certificate not renewed: expires in 23h58m44s
$ step ca renew --force --expires-in 24h foo.crt foo.key
Your certificate has been saved in foo.crt.

I believe this will allow you to use cron, as described, to run at a regular cadence (e.g., every 5 minutes) but only renew if necessary. Note that we do add a jitter to <duration> (a random amount between 0 and <duration>/20). This helps with thundering herd problems if all of your servers try to renew at the same time (this is especially useful if you're using cron).

systemd-based renewal

You can also use systemd for renewal, with the --daemon flag. There are some advantages to this method since the --daemon flag is a little bit more sophisticated: it will automatically schedule renewals at an appropriate cadence based on the certificate lifetime and uses exponential backoff in case of failure.

Here's an example of setting up everything via systemd:

$ cat <<EOF | sudo tee /etc/systemd/system/step.service > /dev/null
[Unit]
Description=Automated certificate management
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=mmalone
ExecStart=/usr/bin/step ca renew --daemon /home/mmalone/foo.crt /home/mmalone/foo.key

[Install]
WantedBy=multi-user.target
EOF

Start the service:

$ sudo systemctl start step

And tell systemd to restart it on reboot:

$ sudo systemctl enable step

[emmetog]

Ah, I was getting tripped up by --renew-period which told me that I should run in --daemon mode. The --expires-in argument is exactly what I was looking for! Thanks for such a detailed response.

I have to admit that I didn't actually check the online docs, just the command line (step ca renew --help) so maybe worth mentioning both cron-based and systemd based renewals there.

$ step --version
Smallstep CLI/0.14.6 (linux/amd64)
Release Date: 2020-07-02 22:15 UTC

Here's the description of the help from my cli, this probably should mention the other renewal methods too:

DESCRIPTION
      step ca renew command renews the given certificate (with a request to the
      certificate authority) and writes the new certificate to disk - either
      overwriting crt-file or using a new file when the --out=file flag is used.

      With the --daemon flag the command will periodically update the given
      certificate. By default, it will renew the certificate before 2/3 of the
      validity period of the certificate has elapsed. A random jitter is used to
      avoid multiple instances running at the same time. The amount of time
      between renewal and certificate expiration can be configured using the
      --expires-in flag, or a fixed period can be set with the --renew-period
      flag.

      The --daemon flag can be combined with --pid, --signal, or --exec to
      provide certificate reloads on your services.

Thanks again!

[mmalone]

just the command line (step ca renew --help) so maybe worth mentioning both cron-based and systemd based renewals there.

Yea, the --expires-in flag is documented (lemme know if you're not seeing it) but there's not a lot of explanation around it. Seems like a quick explanation of one-shot renewal in the description section and an example at the end using expires-in would have been helpful?

I wonder if we should add an example of using systemd and cron in examples, too? Or even a flag that auto-configures systemd and/or cron. The thing I worry about there is that we'll really need to nail some details like file-system location and service start order that, I think, are somewhat OS/distro-dependent.

[dopey]

What about linking out to a tutorial called "automated renewal"? The examples you wrote up could easily be the skeleton for a tutorial page. In that documentation section we could have a line like "See [this tutorial] for some examples of automated certificate renewal."

I just don't know if we want to have https links in our cli docs. On the other hand I don't want to add systemd configuration examples to the cli docs. I think we've done a good job of making sure those docs aren't too bloated so far. But that also means we're missing some useful examples.

[emmetog]

--expires-in is indeed there, it's documented well. I'm guilty of skimming I'm afraid, I saw --renew-period and thought it did what I wanted.

I agree, I'd reckon that just a mention that cron-based and systemd based ways of renewing are both supported would be fine, maybe with a link to check online for more info on each one (although maybe not needed). In my skimming I saw the tool go into lots of detail on --daemon mode and just didn't realize the alternatives.

Maybe the description is not the place for such an extensive explanation of --daemon? Maybe the description should be dedicated to outlining all ways, then describe --daemon in full detail in the explanation of that arg later down?

[dopey]

Going to close this issue in favor of #354 which is more scoped towards improving the documentation.

Thanks again for asking the question and bringing this to our attention.