Open mrhackcz opened 6 years ago
danimo
commented
3 years ago Here is a Gist with a conceptual patch. However, it does not implement some vital aspects, especially
kousu
commented
3 years ago Hello friends :)
I am motivated to figure out some kind of --dry-run so I can wrap dehydrated in an ansible script. It will cover anywhere from 1 to 1000 servers, and I can't, and definitely can't ask my lab to, manually verify the configuration of each server before attempting to get a cert. But letsencrypt's rate limits are set to 5 per week
Renewals are treated specially: they don’t count against your Certificates per Registered Domain limit, but they are subject to a Duplicate Certificate limit of 5 per week. Note: renewals used to count against your Certificate per Registered Domain limit until March 2019, but they don’t anymore. Exceeding the Duplicate Certificate limit is reported with the error message
too many certificates already issued for exact set of domains.
so are buypass's:
We also have a Duplicate Certificate limit of 5 certificates per week.
and I already ran into this once and had to wait a week before I could continue developing.
--dry-run could also be a solution for #792: instead of implementing retry logic, include a dry run step in the regular cronjob: the cronjob will take care of retrying and there's less risk of negative consequences from turning up the cronjob's rate.
I discovered that certbot has
--dry-run Perform a test run of the client, obtaining test (invalid) certificates but not saving them to disk. This can currently only be used with the 'certonly' and 'renew' subcommands. Note: Although --dry-run tries to avoid making any persistent changes on a system, it is not completely side-effect free: if used with webserver authenticator plugins like apache and nginx, it makes and then reverts temporary config changes in order to obtain test certificates, and reloads webservers to deploy and then roll back those changes. It also calls --pre-hook and --post-hook commands if they are defined because they may be necessary to accurately simulate renewal. --deploy- hook commands are not called. (default: False)
So with certbot, I think they expect that if you're at all unsure about your configuration (in DNS, webserver, domains, connectivity) you should always do:
certbot renew --dry-run && certbot renewwhich:
letsencrypt-test to try to make new certsletsencrypt to make new ones, burning one of your attempts for the weekcertbot, configuration is the same as getting an initial cert (certbot certonly -d "$(hostname) [and probably some other options]")To do a proper dry run, you must contact a server. I just wish I could do that dry run without And now with multiple ACME servers in the wild it's complicated:
--ca letsencrypt vs --ca letsencrypt-test--ca buypass vs --ca buypass-test--ca zerossl (which doesn't have a staging server; but it also doesn't have rate limits, so maybe we can consider it its own staging server)
Then I tried@danimo's patch doesn't seem to address this? Maybe it was written before multiple ACME CAs existed?
dehydrated-based workarounds/extensionsdehydrated --ca testencrypt-test -c && dehydrated -cwhich will
letsencrypt-test to get new certs, replacing the working certs with test certsSo this is closer, but leaves me with invalid test certs on my server.
This form:
dehydrated --ca testencrypt-test -c && (eval "$(dehydrated -e)"; rm "$CERTDIR"/*/cert.pem) && dehydrated -cletsencrypt-test to get new certs, replacing the working certs with test certsDestryoing all certs is okay for my situation, as I am only planning one domain per server at the moment, but it's a dangerous precedent to set.
This form:
dehydrated --ca testencrypt-test -o $(mktemp -d) -c && dehydrated -cletsencrypt-test to try to make new certs, and then forgets themletsencrypt to get new certs, burning one of your attempts for the weekHowever, this wastes about 10s per deploy, by not being smart enough.
So the previous two forms both have the right result, at the expense of a. danger or b. time.
Maybe a hook could do it?
cat > /usr/local/bin/dehydrated/dehydrated-dry-run.sh <<EOF && chmod +x /usr/local/bin/dehydrated/dehydrated-dry-run.sh
#!/bin/sh
# a dehydrated hook script that deletes newly generated certs
# emulates `certbot --dry-run`
# see https://github.com/dehydrated-io/dehydrated/issues/561
deploy_cert() {
local DOMAIN="${1}" KEYFILE="${2}" CERTFILE="${3}" FULLCHAINFILE="${4}" CHAINFILE="${5}" TIMESTAMP="${6}"
rm "${CERTFILE}" "${KEYFILE}" "${CHAINFILE}" "${FULLCHAINFILE}"
}
HANDLER="$1"; shift
if [[ "${HANDLER}" =~ ^(deploy_cert)$ ]]; then
"$HANDLER" "$@"
fi
EOFdehydrated --ca testencrypt-test --hook /usr/local/bin/dehydrated/dehydrated-dry-run.sh -c && dehydrated -cThis seems to check all my boxes:
and it is idempotent, only doing work when work is needed.
lukas2511
commented
3 years ago @kousu Mh, from what I understand you are suggesting that a "dry-run" would connect to a staging environment of the corresponding CA? My understanding of a dry-run would be something that involves no real connections and only internal / partially mocked data. The staging environments have API limits too, so using them as some form of dry-run would only postpone eventual rate limit issues.
My big goal (for which I unfortunately didn't find much time yet...) is to restructure dehydrated into smaller internal parts, making it easier to change overall execution flow and replace parts in the future. That would make it a lot easier to implement real dry-runs, would allow for more flexibility with e.g. different CAs for different certificates, CA specific weirdness, etc.
kousu
commented
3 years ago That's right. I want to be able to vet my setup: DNS, file permissions, webserver, ACME configuration files. Some of these can be tricky the first time, and others can get changed over time outside of my control. The only way to be sure is to run the complete ACME protocol.
The rate limits on the staging servers are much higher and designed to support this sort of testing and development.
Maybe --dry-run is the wrong name for the feature I'm looking for. Maybe I want --test and I should start a separate issue for it. --ca letsencrypt-test goes most of the way to what I want, I just wish it wouldn't overwrite my existing certs on success, because then I could automate `` and be pretty confident that I can detect and fix a bad configuration long before getting banned.
Right now I either run
dehydrated -cwhich got me banned last time
dehydrated -c --ca letsencrypt-test && dehydrated -cwhich doesn't get me banned but leaves me with invalid certs or
dehydrated -c --ca letsencrypt-test -o $(mktemp -d) && dehydrated -cwhich gets me valid certs, but spends an extra 10s per attempt that it really doesn't need to.
The ideal for me would be a --test that didn't save certs to disk so that
dehydrated -c --ca letsencrypt-test --test && dehydrated -cwould only run when certs are expiring, would still produce valid certs, and would avoid getting banned all at the same time.
My big goal (for which I unfortunately didn't find much time yet...)
Oh this is the story of my life too! No worries. We're all volunteers here.
jasoncodes
commented
3 years ago dehydrated -c --ca letsencrypt-test -o $(mktemp -d) && dehydrated -cwhich gets me valid certs, but spends an extra 10s per attempt that it really doesn't need to.
Have you tried copying your current real certs over to a temporary directory before running against letsencrypt-test?
Something like this should avoid requesting certs from the test CA unnecessarily:
rsync --archive /etc/dehydrated/certs{,.tmp}/ &&
dehydrated -c --ca letsencrypt-test -o /etc/dehydrated/certs.tmp &&
rm -r /etc/dehydrated/certs.tmp &&
dehydrated -c@jasoncodes it's genius. I think I will use it! Thank you.
kousu
commented
3 years ago For what it's worth, certbot decided that their --dry-run implies contacting the staging server; their --dry-run ~= --ca letsencrypt-test -o $(mktemp -d). -v --dryrun reports:
Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
I take back what I said above: their --dry-run runs regardless of the state of your actual certs. So
certbot renew --dry-run && certbot renewis functionally identical to
dehydrated --ca testencrypt-test -o $(mktemp -d) -c && dehydrated -cWhich is probably good enough for my purposes, honestly. But Jason's solution is better :)
kousu
commented
3 years ago @jasoncodes I've refined your suggestion into:
(
# create/renew certs if necessary, while doing a dry-run to first verify the configuration
# the dry-run helps avoid hitting https://letsencrypt.org/docs/rate-limits/.
# see also https://github.com/dehydrated-io/dehydrated/issues/561
set -eu
eval "$(dehydrated --env)" # load $CERTDIR
TMP_CERTDIR="$(mktemp -d)"
trap 'rm -r "$TMP_CERTDIR"' EXIT # beware: bashism
rsync -a "$CERTDIR"/ "$TMP_CERTDIR"
dehydrated -c --ca letsencrypt-test -o "$TMP_CERTDIR" >/dev/null # dry-run
dehydrated -c
)It's working well so far :)
Can you please to the next version add a option to run "dry-run" ?