AWS key not being picked up

[shanedobson-ibboost]

I run several automated cert renewals in Jenkins using the DNS method. They've been running fine for a few months now. But suddenly, just a couple of them are failing, with an error about the AWS key ID and secret key.

These are both definitely exported and have worked in the past, and are currently working for other jobs, just not for these two.

Steps to reproduce

08:40:42 SSH: EXEC: STDOUT/STDERR from command [/home/jenkins/.acme.sh/acme.sh --cron --dns dns_aws -d mydomain.com] ...
08:40:42 [Mon Jan 15 08:40:42 GMT 2018] ===Starting cron===
08:40:42 [Mon Jan 15 08:40:42 GMT 2018] Renew: 'mydomain.com'
08:40:42 [Mon Jan 15 08:40:42 GMT 2018] Single domain='mydomain.com'
08:40:42 [Mon Jan 15 08:40:42 GMT 2018] Getting domain auth token for each domain
08:40:42 [Mon Jan 15 08:40:42 GMT 2018] Getting webroot for domain='mydomain.com'
08:40:42 [Mon Jan 15 08:40:42 GMT 2018] Getting new-authz for domain='mydomain.com'
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] The new-authz request is ok.
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] Found domain api file: /home/jenkins/.acme.sh/dnsapi/dns_aws.sh
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] You don't specify aws route53 api key id and and api key secret yet.
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] Please create you key and try again. see https://github.com/Neilpang/acme.sh/wiki/How-to-use-Amazon-Route53-API
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] Error add txt for domain:_acme-challenge.mydomain.com
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] Please add '--debug' or '--log' to check more details.
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] Error renew mydomain.com.
08:40:45 [Mon Jan 15 08:40:45 GMT 2018] ===End cron===

Debug log

[jenkins@don user]$ /home/jenkins/.acme.sh/acme.sh --cron --dns dns_aws -d mydomain.com --debug [Mon Jan 15 08:54:17 GMT 2018] Lets find script dir. [Mon Jan 15 08:54:17 GMT 2018] SCRIPT='/home/jenkins/.acme.sh/acme.sh' [Mon Jan 15 08:54:17 GMT 2018] _script='/home/jenkins/.acme.sh/acme.sh' [Mon Jan 15 08:54:17 GMT 2018] _script_home='/home/jenkins/.acme.sh' [Mon Jan 15 08:54:17 GMT 2018] Using config home:/home/jenkins/.acme.sh https://github.com/Neilpang/acme.sh v2.6.9 [Mon Jan 15 08:54:17 GMT 2018] Using config home:/home/jenkins/.acme.sh [Mon Jan 15 08:54:17 GMT 2018] ===Starting cron=== [Mon Jan 15 08:54:17 GMT 2018] Using config home:/home/jenkins/.acme.sh [Mon Jan 15 08:54:17 GMT 2018] _stopRenewOnError [Mon Jan 15 08:54:17 GMT 2018] di='/home/jenkins/.acme.sh/mydomain.com/' [Mon Jan 15 08:54:17 GMT 2018] d='mydomain.com' [Mon Jan 15 08:54:17 GMT 2018] Using config home:/home/jenkins/.acme.sh [Mon Jan 15 08:54:17 GMT 2018] DOMAIN_PATH='/home/jenkins/.acme.sh/mydomain.com' [Mon Jan 15 08:54:17 GMT 2018] Renew: 'mydomain.com' [Mon Jan 15 08:54:17 GMT 2018] Using config home:/home/jenkins/.acme.sh [Mon Jan 15 08:54:17 GMT 2018] Using api: https://acme-v01.api.letsencrypt.org [Mon Jan 15 08:54:17 GMT 2018] Le_NextRenewTime='1513940063' [Mon Jan 15 08:54:17 GMT 2018] _on_before_issue [Mon Jan 15 08:54:17 GMT 2018] Le_LocalAddress [Mon Jan 15 08:54:17 GMT 2018] Check for domain='mydomain.com' [Mon Jan 15 08:54:17 GMT 2018] _currentRoot='dns_aws' [Mon Jan 15 08:54:17 GMT 2018] _saved_account_key_hash is not changed, skip register account. [Mon Jan 15 08:54:17 GMT 2018] Read key length: [Mon Jan 15 08:54:17 GMT 2018] _createcsr [Mon Jan 15 08:54:17 GMT 2018] Single domain='mydomain.com' [Mon Jan 15 08:54:17 GMT 2018] Getting domain auth token for each domain [Mon Jan 15 08:54:17 GMT 2018] Getting webroot for domain='mydomain.com' [Mon Jan 15 08:54:18 GMT 2018] _w='dns_aws' [Mon Jan 15 08:54:18 GMT 2018] _currentRoot='dns_aws' [Mon Jan 15 08:54:18 GMT 2018] Getting new-authz for domain='mydomain.com' [Mon Jan 15 08:54:18 GMT 2018] Try new-authz for the 0 time. [Mon Jan 15 08:54:18 GMT 2018] url='https://acme-v01.api.letsencrypt.org/acme/new-authz' [Mon Jan 15 08:54:18 GMT 2018] payload='{"resource": "new-authz", "identifier": {"type": "dns", "value": "mydomain.com"}}' [Mon Jan 15 08:54:18 GMT 2018] RSA key [Mon Jan 15 08:54:19 GMT 2018] GET [Mon Jan 15 08:54:19 GMT 2018] url='https://acme-v01.api.letsencrypt.org/directory' [Mon Jan 15 08:54:19 GMT 2018] timeout [Mon Jan 15 08:54:19 GMT 2018] _CURL='curl -L --silent --dump-header /home/jenkins/.acme.sh/http.header ' [Mon Jan 15 08:54:19 GMT 2018] ret='0' [Mon Jan 15 08:54:19 GMT 2018] POST [Mon Jan 15 08:54:19 GMT 2018] url='https://acme-v01.api.letsencrypt.org/acme/new-authz' [Mon Jan 15 08:54:19 GMT 2018] _CURL='curl -L --silent --dump-header /home/jenkins/.acme.sh/http.header ' [Mon Jan 15 08:54:21 GMT 2018] _ret='0' [Mon Jan 15 08:54:21 GMT 2018] code='201' [Mon Jan 15 08:54:21 GMT 2018] The new-authz request is ok. [Mon Jan 15 08:54:21 GMT 2018] entry='"type":"dns-01","status":"pending","uri":"https://acme-v01.api.letsencrypt.org/acme/challenge/IazjoXBe3_Z2YBGSfv8VvIu8_3urpTCOTSSu9ZKuTns/3101257922","token":"Llkjidsf908734215;kljhadmVbhjH518f634"' [Mon Jan 15 08:54:21 GMT 2018] token='LFV-98u3145t;ijad[ouoadgfa34' [Mon Jan 15 08:54:21 GMT 2018] uri='https://acme-v01.api.letsencrypt.org/acme/challenge/IazjoXBe3_Z2YBGSfv8VvIu8_3urpTCOTSSu9ZKuTns/3101257922' [Mon Jan 15 08:54:21 GMT 2018] keyauthorization='LFVwer657htjXafweqreuefenWOPiFasdfasdfasdfNUATP-8u349-08uaepoiujt9jqergyBw' [Mon Jan 15 08:54:21 GMT 2018] dvlist='mydomain.com#LFVSsfghsfghfgh18f634.MQPpimrjN_P3eaPrpLxcNUATPnHEuqgRs76CMXmKyBw#https://acme-v01.api.letsencrypt.org/acme/challenge/IazjoXB-98u34jrt-9japoijqawefqewrfe3_Z2YBGSfv8VvIu8_3urpTCOTSSu9ZKuTns/3101257922#dns-01#dns_aws' [Mon Jan 15 08:54:21 GMT 2018] vlist='mydomain.com#LFVS5TOmXeuefenWOPiFhHi5z00v78oyuiyr467kfkryuiert634.MQPpimrjN_P3eaPrpLxcNUATPnHEuqgRs76CMXmKyBw#https://acme-v01.api.letsencrypt.org/acme/challenge/Iazjoweroije9-89j134598j9we8jtrqwertXBe3_Z2YBGSfv8VvIu8_3urpTCOTSSu9ZKuTns/3101257922#dns-01#dns_aws,' [Mon Jan 15 08:54:21 GMT 2018] txtdomain='_acme-challenge.mydomain.com' [Mon Jan 15 08:54:21 GMT 2018] txt='5V2adwjkfr-9uj459086j-98jqer[oertj-q98jqrt-9j54tAKFqg' [Mon Jan 15 08:54:21 GMT 2018] d_api='/home/jenkins/.acme.sh/dnsapi/dns_aws.sh' [Mon Jan 15 08:54:21 GMT 2018] Found domain api file: /home/jenkins/.acme.sh/dnsapi/dns_aws.sh [Mon Jan 15 08:54:21 GMT 2018] You don't specify aws route53 api key id and and api key secret yet. [Mon Jan 15 08:54:21 GMT 2018] Please create you key and try again. see https://github.com/Neilpang/acme.sh/wiki/How-to-use-Amazon-Route53-API [Mon Jan 15 08:54:21 GMT 2018] Error add txt for domain:_acme-challenge.mydomain.com [Mon Jan 15 08:54:21 GMT 2018] pid [Mon Jan 15 08:54:21 GMT 2018] No need to restore nginx, skip. [Mon Jan 15 08:54:21 GMT 2018] _clearupdns [Mon Jan 15 08:54:21 GMT 2018] Dns not added, skip. [Mon Jan 15 08:54:21 GMT 2018] _on_issue_err [Mon Jan 15 08:54:21 GMT 2018] Please add '--debug' or '--log' to check more details. [Mon Jan 15 08:54:21 GMT 2018] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh [Mon Jan 15 08:54:21 GMT 2018] Diagnosis versions: openssl:openssl OpenSSL 1.0.1e-fips 11 Feb 2013 apache: apache doesn't exists. nc: usage: nc [-46DdhklnrStUuvzC] [-i interval] [-p source_port] [-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_version] [-x proxy_address[:port]] [hostname] [port[s]] Command Summary: -4 Use IPv4 -6 Use IPv6 -D Enable the debug socket option -d Detach from stdin -h This help text -i secs Delay interval for lines sent, ports scanned -k Keep inbound sockets open for multiple connects -l Listen mode, for inbound connects -n Suppress name/port resolutions -p port Specify local port for remote connects -r Randomize remote ports -S Enable the TCP MD5 signature option -s addr Local source address -T ToS Set IP Type of Service -C Send CRLF as line-ending -t Answer TELNET negotiation -U Use UNIX domain socket -u UDP mode -v Verbose -w secs Timeout for connects and final net reads -X proto Proxy protocol: "4", "5" (SOCKS) or "connect" -x addr[:port] Specify proxy address and port -z Zero-I/O mode [used for scanning] Port numbers can be individual or ranges: lo-hi [inclusive] [Mon Jan 15 08:54:21 GMT 2018] Return code: 1 [Mon Jan 15 08:54:21 GMT 2018] Error renew mydomain.com. [Mon Jan 15 08:54:21 GMT 2018] ===End cron===

I can't figure this out.

[Neilpang]

check:

whoami

pwd

crontab -l

cat  /home/jenkins/.acme.sh/account.conf

[Neilpang]

And:

/home/jenkins/.acme.sh/acme.sh --cron   --home /home/jenkins/.acme.sh

[shanedobson-ibboost]

[jenkins@don user]$ whoami
jenkins
[jenkins@don user]$ pwd
/home/user
[jenkins@don user]$ crontab -l

[jenkins@don user]$ cat  /home/jenkins/.acme.sh/account.conf

#LOG_FILE="/home/jenkins/.acme.sh/acme.sh.log"
#LOG_LEVEL=1

#AUTO_UPGRADE="1"

#NO_TIMESTAMP=1

USER_PATH='/sbin:/bin:/usr/sbin:/usr/bin:/var/rudder/cfengine-community/bin:/var/rudder/cfengine-community/bin'
[jenkins@don user]$ /home/jenkins/.acme.sh/acme.sh --cron   --home /home/jenkins/.acme.sh
[Mon Jan 15 10:00:37 GMT 2018] ===Starting cron===
[Mon Jan 15 10:00:37 GMT 2018] Renew: 'mydomain.com'
[Mon Jan 15 10:00:37 GMT 2018] Single domain='mydomain.com'
[Mon Jan 15 10:00:37 GMT 2018] Getting domain auth token for each domain
[Mon Jan 15 10:00:37 GMT 2018] Getting webroot for domain='mydomain.com'
[Mon Jan 15 10:00:37 GMT 2018] Getting new-authz for domain='mydomain.com'
[Mon Jan 15 10:00:41 GMT 2018] The new-authz request is ok.
[Mon Jan 15 10:00:41 GMT 2018] Found domain api file: /home/jenkins/.acme.sh/dnsapi/dns_aws.sh
[Mon Jan 15 10:00:41 GMT 2018] You don't specify aws route53 api key id and and api key secret yet.
[Mon Jan 15 10:00:41 GMT 2018] Please create you key and try again. see https://github.com/Neilpang/acme.sh/wiki/How-to-use-Amazon-Route53-API
[Mon Jan 15 10:00:41 GMT 2018] Error add txt for domain:_acme-challenge.mydomain.com
[Mon Jan 15 10:00:41 GMT 2018] Please add '--debug' or '--log' to check more details.
[Mon Jan 15 10:00:41 GMT 2018] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh
[Mon Jan 15 10:00:41 GMT 2018] Error renew mydomain.com.
[Mon Jan 15 10:00:41 GMT 2018] ===End cron===

[FernandoMiguel]

can you upgrade to the lastest version of acme.sh , please? acme.sh --upgrade should bring you to v2.7.5

also, can you list make sure that with that current aws key, you can still list the records for that zoneid? From https://docs.aws.amazon.com/cli/latest/reference/route53/list-resource-record-sets.html $aws route53 list-resource-record-sets --hosted-zone-id

thanks

[Neilpang]

try:

 cat  ~/.acme.sh/account.conf

[FernandoMiguel]

careful not to expose sensitive information like your long lived access keys :)

[shanedobson-ibboost]

[jenkins@don ibb_admin]$  cat  ~/.acme.sh/account.conf

#LOG_FILE="/home/jenkins/.acme.sh/acme.sh.log"
#LOG_LEVEL=1

#AUTO_UPGRADE="1"

#NO_TIMESTAMP=1

USER_PATH='/sbin:/bin:/usr/sbin:/usr/bin:/var/rudder/cfengine-community/bin:/var/rudder/cfengine-community/bin'

[shanedobson-ibboost]

@FernandoMiguel thanks for reminding me of that, though as far as I can tell I didn't actually expose those, but I've amended some info in the original question.

[FernandoMiguel]

it's darn easy to push an AWS key... and then you have to go to the trouble of rotating everything. if you are running jenkins on aws EC2, you might consider AWS EC2 Roles instead of keys in env files/vars

[shanedobson-ibboost]

Hmm, I don't seem to be able to get that command to run, but I did upgrade acme.sh.

[jenkins@don ibb_admin]$ aws route53 list-resource-record-sets --hosted-zone-id
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument --hosted-zone-id: expected one argument

[shanedobson-ibboost]

That's a good point about the key. I'm a little short on time at the moment and really need to get these certs renewed, but I think the approach you suggest makes a lot of sense.

[FernandoMiguel]

i assume you are entering your zoneid in front of --hosted-zone-id ? somehow github web ate my example zone id from the previous comment lol sorry about that

aws route53 list-resource-record-sets --hosted-zone-id Z2LD58HEXAMPLE

[shanedobson-ibboost]

Apologies for my noobness, I was not. Running that command revealed the AWS credentials has been wiped out. I reentered those and could list the DNS records successfully with that command. But the renewal is still failing with the same error.

[FernandoMiguel]

maybe the envvar for jenkins starting acme.sh arent there ? not sure how you are passing those around. also be extremely careful to not exceed LE API limit or you wont be able to renew for a week. use --staging until you got it working and then switch back to production

[shanedobson-ibboost]

Thanks for the tip about staging. I'm really not sure what the problem is or why it has suddenly appeared. But I'll keep working on it and will respond again when I resolve it.

[FernandoMiguel]

my suspicion is that there are two problems one probably related with the aws access key. the other related to an expired TXT record.... before was working cause it was still there

[shanedobson-ibboost]

Spent days on this and found no resolution. Essentially, acme.sh just kept telling me that AWS credentials weren't configured for editing the DNS records. But they were, they key was correct and the user definitely has the right permissions (its used on other servers too). In the end, I ran out of of time and had no choice but to issue new certs using certbot. This will buy me a couple of months.

[FernandoMiguel]

@shanedobson-ibboost remind me how you are applying the access keys to acme.sh pls

[shanedobson-ibboost]

[jenkins@server]$ aws configure
AWS Access Key ID [****************KJJG]:
AWS Secret Access Key [****************uwyr]:
Default region name [eu-west-1]:
Default output format [None]:

[FernandoMiguel]

maybe i'm wrong , but i dont think acme.sh picks the keys from ~/aws/credentials

they need to be in env vars

from https://github.com/Neilpang/acme.sh/tree/master/dnsapi#10-use-amazon-route53-domain-api export AWS_ACCESS_KEY_ID=XXXXXXXXXX export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXX

[shanedobson-ibboost]

Man, you were correct @FernandoMiguel - thank you so much for your help, that was incredible frustrating. I exported them as you suggested and now the renewals are working.

I don't understand why this was working and then stopped, as in the credentials disappeared from the server randomly. However it's working now so thanks to you both.

[FernandoMiguel]

i suspect you exported them when you first generated the certs, and at somepoint restarted the server or flushed the vars

[shanedobson-ibboost]

So if the server is restarted will this flush the vars? I'll need to do some research to make those persist then.

[FernandoMiguel]

i think they get saved to a conf file.... but i also know @Neilpang recently changed that if you are using STS

so ¯_(ツ)_/¯

[shanedobson-ibboost]

I just added the information to /etc/environment and that seems to be doing the trick.