search

gregtwallace / certwarden

Cert Warden is a centralized ACME Client. It provides an API for certificate consumers to fetch their individual keys and certs with API keys.
https://www.certwarden.com/
Other
176 stars 6 forks source link

Difficulties with Route 53 #47

Closed sapid closed 3 months ago

sapid commented 3 months ago

My current goal is to create a wildcard SSL cert on a network behind a NAT. As such, I'm trying to use the DNS-01 challenge method using an API for Route 53. I'm using a root-less installation, and I have legocerthub running on linux (not in Docker) using the provided installation script (scripts/install.sh) with the caveat that I did not grant CAP_NET_BIND_SERVICE to legocerthub since I don't plan on exposing a port through the NAT for the http challenge (in spite of the headache it would save me).

I've successfully used both lego and acme.sh to accomplish cert issuance with the Let's Encrypt staging ACME server with my credentials for Route 53, but I get errors in legocerthub when I use either Providers dns01acmesh or dns01goacme. All of these experiments below are performed from the same machine.

dns01acmesh

In the case of dns01acmesh, it looks to me like the root cause is that something is preventing acme.sh from reading from or writing to its config file. I get this error when acme.sh's path is specific as /opt/legocerthub/.acme.sh and as /opt/legocerthub/scripts/acme.sh. (The write error could be a red herring if not saving the config is the intended outcome, but as I mentioned above, the same tokens specified here work from the cli with both tools, and they look correct in data/app/config.yaml, so the alternative appears perhaps to be passing the credentials to acme.sh incorrectly.)

The valid CLI invocation looks like:

./acme.sh --install --accountemail my@valid.email
./acme.sh --set-default-ca --server letsencrypt_test
export AWS_ACCESS_KEY_ID="MY_VALID_KEY"
export AWS_SECRET_ACCESS_KEY="MY_VALID_SECRET_KEY"
export AWS_HOSTED_ZONE_ID="MY_ZONE_ID"
./acme.sh --issue --dns dns_aws --keylength 4096 -d *.int.my_valid_domain.com

Config for dns01acmesh:

Relevant log output

After successfully fetching the challenge, here's what the logs look like. I wish the debug log level showed how acme.sh is being invoked.

{"level":"debug","ts":"2024-03-21T18:44:00.491-0700","caller":"challenges/provisioning.go:25","msg":"added resource for int.my_valid_domain.com to challenge work tracker"},
{"level":"error","ts":"2024-03-21T18:44:04.014-0700","caller":"dns01acmesh/resources.go:24","msg":"acme.sh dns create script std err: [Thu Mar 21 18:44:00 PDT 2024] config file is empty, can not save SAVED_AWS_ACCESS_KEY_ID=\"VALID_ACCESS_KEY\"\n[Thu Mar 21 18:44:00 PDT 2024] config file is empty, can not clear\n[Thu Mar 21 18:44:00 PDT 2024] config file is empty, can not save SAVED_AWS_SECRET_ACCESS_KEY=\"VALID_SECRET_KEY\"\n[Thu M
ar 21 18:44:00 PDT 2024] config file is empty, can not clear\n[Thu Mar 21 18:44:00 PDT 2024] config file is empty, can not save SAVED_AWS_DNS_SLOWRATE=\"1\"\n[Thu Mar 21 18:44:00 PDT 2024] config file is empty, can not clear\n[Thu M
ar 21 18:44:01 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<ErrorResponse xmlns=\"https://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The security token included
in the request is invalid.</Message></Error><RequestId>98d9a449-0b8d-4a5f-a47d-272c5f3c2bc7</RequestId></ErrorResponse>\n[Thu Mar 21 18:44:01 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<ErrorResponse xmlns=\"https://route53.a
mazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The security token included in the request is invalid.</Message></Error><RequestId>f7759888-d650-4788-9fcd-0f488a936df2</RequestId></
ErrorResponse>\n[Thu Mar 21 18:44:02 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<ErrorResponse xmlns=\"https://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The se
curity token included in the request is invalid.</Message></Error><RequestId>2192c14f-087d-4d64-af3e-95e7ebfe4f72</RequestId></ErrorResponse>\n[Thu Mar 21 18:44:02 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<ErrorResponse xml
ns=\"https://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The security token included in the request is invalid.</Message></Error><RequestId>b8b29fc5-b33d-4b5b-8b9b-8d30
aaafd451</RequestId></ErrorResponse>\n[Thu Mar 21 18:44:02 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<ErrorResponse xmlns=\"https://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId
</Code><Message>The security token included in the request is invalid.</Message></Error><RequestId>7dd23a28-059f-492d-aaa4-a1692fab2308</RequestId></ErrorResponse>\n/opt/legocerthub/.acme.sh/temp/acme.sh_dns_aws.sh: line 8134: _erro
r: command not found\n[Thu Mar 21 18:44:03 PDT 2024] invalid domain\n","stacktrace":"legocerthub-backend/pkg/challenges/providers/dns01acmesh.(*Service).Provision\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/providers
/dns01acmesh/resources.go:24\nlegocerthub-backend/pkg/challenges.(*Service).provision\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/provisioning.go:60\nlegocerthub-backend/pkg/challenges.(*Service).Solve\n\t/home/runne
r/work/legocerthub/legocerthub/pkg/challenges/solver.go:55\nlegocerthub-backend/pkg/domain/authorizations.(*Service).authWorker\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:132\nlegocerthub-bac
kend/pkg/domain/authorizations.(*Service).fulfillAuth\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:106\nlegocerthub-backend/pkg/domain/authorizations.(*Service).FulfillAuths.func1\n\t/home/runn
er/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:28"},
{"level":"error","ts":"2024-03-21T18:44:04.016-0700","caller":"dns01acmesh/resources.go:27","msg":"acme.sh dns create script error: exit status 1","stacktrace":"legocerthub-backend/pkg/challenges/providers/dns01acmesh.(*Service).Pro
vision\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/providers/dns01acmesh/resources.go:27\nlegocerthub-backend/pkg/challenges.(*Service).provision\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/provisioni
ng.go:60\nlegocerthub-backend/pkg/challenges.(*Service).Solve\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/solver.go:55\nlegocerthub-backend/pkg/domain/authorizations.(*Service).authWorker\n\t/home/runner/work/legocer
thub/legocerthub/pkg/domain/authorizations/fulfiller.go:132\nlegocerthub-backend/pkg/domain/authorizations.(*Service).fulfillAuth\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:106\nlegocerthub-b
ackend/pkg/domain/authorizations.(*Service).FulfillAuths.func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:28"},
{"level":"error","ts":"2024-03-21T18:44:07.475-0700","caller":"dns01acmesh/resources.go:49","msg":"acme.sh dns create script std err: [Thu Mar 21 18:44:04 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<ErrorResponse xmlns=\"http
s://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The security token included in the request is invalid.</Message></Error><RequestId>956f1066-37c3-419f-86a0-ec5d6d757957<
/RequestId></ErrorResponse>\n[Thu Mar 21 18:44:04 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<ErrorResponse xmlns=\"https://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><M
essage>The security token included in the request is invalid.</Message></Error><RequestId>70b0fb60-e824-46ed-939c-5f5374d08d04</RequestId></ErrorResponse>\n[Thu Mar 21 18:44:05 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<Erro
rResponse xmlns=\"https://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The security token included in the request is invalid.</Message></Error><RequestId>2c0db589-469a-4
6bc-a4d8-ad56fa4909fc</RequestId></ErrorResponse>\n[Thu Mar 21 18:44:05 PDT 2024] Response error:<?xml version=\"1.0\"?>\n<ErrorResponse xmlns=\"https://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>Invalid
ClientTokenId</Code><Message>The security token included in the request is invalid.</Message></Error><RequestId>913a5ec8-a350-4a17-8647-bd5088a44015</RequestId></ErrorResponse>\n[Thu Mar 21 18:44:06 PDT 2024] Response error:<?xml ve
rsion=\"1.0\"?>\n<ErrorResponse xmlns=\"https://route53.amazonaws.com/doc/2013-04-01/\"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The security token included in the request is invalid.</Message></Error><Req
uestId>fd9603b5-2b0b-47f3-b490-cba7f6960e60</RequestId></ErrorResponse>\n/opt/legocerthub/.acme.sh/temp/acme.sh_dns_aws.sh: line 8134: _error: command not found\n[Thu Mar 21 18:44:06 PDT 2024] invalid domain\n","stacktrace":"legocer
thub-backend/pkg/challenges/providers/dns01acmesh.(*Service).Deprovision\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/providers/dns01acmesh/resources.go:49\nlegocerthub-backend/pkg/challenges.(*Service).deprovision\n\
t/home/runner/work/legocerthub/legocerthub/pkg/challenges/provisioning.go:91\nlegocerthub-backend/pkg/challenges.(*Service).Solve.func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/solver.go:63\nlegocerthub-backend/pk
g/challenges.(*Service).Solve\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/solver.go:71\nlegocerthub-backend/pkg/domain/authorizations.(*Service).authWorker\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/auth
orizations/fulfiller.go:132\nlegocerthub-backend/pkg/domain/authorizations.(*Service).fulfillAuth\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:106\nlegocerthub-backend/pkg/domain/authorizations
.(*Service).FulfillAuths.func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:28"},
{"level":"error","ts":"2024-03-21T18:44:07.477-0700","caller":"dns01acmesh/resources.go:52","msg":"acme.sh dns delete script error: exit status 1","stacktrace":"legocerthub-backend/pkg/challenges/providers/dns01acmesh.(*Service).Dep
rovision\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/providers/dns01acmesh/resources.go:52\nlegocerthub-backend/pkg/challenges.(*Service).deprovision\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/provis
ioning.go:91\nlegocerthub-backend/pkg/challenges.(*Service).Solve.func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/solver.go:63\nlegocerthub-backend/pkg/challenges.(*Service).Solve\n\t/home/runner/work/legocerthub/l
egocerthub/pkg/challenges/solver.go:71\nlegocerthub-backend/pkg/domain/authorizations.(*Service).authWorker\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:132\nlegocerthub-backend/pkg/domain/auth
orizations.(*Service).fulfillAuth\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:106\nlegocerthub-backend/pkg/domain/authorizations.(*Service).FulfillAuths.func1\n\t/home/runner/work/legocerthub/
legocerthub/pkg/domain/authorizations/fulfiller.go:28"},
{"level":"debug","ts":"2024-03-21T18:44:07.478-0700","caller":"challenges/provisioning.go:86","msg":"removed resource for int.mydomain.com from challenge work tracker"},
{"level":"error","ts":"2024-03-21T18:44:07.478-0700","caller":"challenges/solver.go:65","msg":"challenge solver deprovision failed (exit status 1)","stacktrace":"legocerthub-backend/pkg/challenges.(*Service).Solve.func1\n\t/home/run
ner/work/legocerthub/legocerthub/pkg/challenges/solver.go:65\nlegocerthub-backend/pkg/challenges.(*Service).Solve\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/solver.go:71\nlegocerthub-backend/pkg/domain/authorization
s.(*Service).authWorker\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:132\nlegocerthub-backend/pkg/domain/authorizations.(*Service).fulfillAuth\n\t/home/runner/work/legocerthub/legocerthub/pkg/d
omain/authorizations/fulfiller.go:106\nlegocerthub-backend/pkg/domain/authorizations.(*Service).FulfillAuths.func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:28"},

dns01goacme

cli execution

aws configure
# configure key, secret key, region interactively
export AWS_HOSTED_ZONE_ID="MY_ZONE_ID"
./lego -s https://acme-staging-v02.api.letsencrypt.org/directory --dns route53 -d *.int.my_valid_domain.com -a --email my@valid.email --dns.resolvers 8.8.8.8 run

Config for dns01goacme

Relevant log output

This one's a bit more puzzling to me, though my (perhaps naive) guess is that it's trying to query EC2 IMDS at 169.254.169.254 instead of directly using the provided credentials with route53.amazonaws.com. (Trying to verify the credentials with IMDS this way will fail in my idiosyncratic case - the IMDS for my Route 53 AWS account and my network AWS account are disjoint.)

{"level":"debug","ts":"2024-03-21T19:30:44.877-0700","caller":"challenges/provisioning.go:25","msg":"added resource for int.my_valid_domain.com to challenge work tracker"},
{"level":"debug","ts":"2024-03-21T19:30:44.889-0700","caller":"challenges/provisioning.go:86","msg":"removed resource for int.my_valid_domain.com from challenge work tracker"},
{"level":"error","ts":"2024-03-21T19:30:44.890-0700","caller":"challenges/solver.go:65","msg":"challenge solver deprovision failed (route53: operation error Route 53: ListResourceRecordSets, failed to sign request: failed to retriev
e credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed)","stacktrace":"legocerthub-backend/pkg/challenges.(*
Service).Solve.func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/solver.go:65\nlegocerthub-backend/pkg/challenges.(*Service).Solve\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/solver.go:71\nlegocerthub
-backend/pkg/domain/authorizations.(*Service).authWorker\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:132\nlegocerthub-backend/pkg/domain/authorizations.(*Service).fulfillAuth\n\t/home/runner/w
ork/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:106\nlegocerthub-backend/pkg/domain/authorizations.(*Service).FulfillAuths.func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:
28"},
{"level":"error","ts":"2024-03-21T19:30:44.890-0700","caller":"orders/fulfilling_do.go:101","msg":"order fulfilling worker 1: fulfill auths error: %!w(*fmt.wrapError=&{route53: operation error Route 53: ListResourceRecordSets, faile
d to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed 0xc000735080})","
stacktrace":"legocerthub-backend/pkg/domain/orders.(*orderFulfillJob).Do\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/orders/fulfilling_do.go:101\nlegocerthub-backend/pkg/datatypes/job_manager.(*Manager[...]).doJob\n\t/ho
me/runner/work/legocerthub/legocerthub/pkg/datatypes/job_manager/do.go:22\nlegocerthub-backend/pkg/datatypes/job_manager.NewManager[...].func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/datatypes/job_manager/manager.go:77"},
{"level":"info","ts":"2024-03-21T19:30:44.896-0700","caller":"orders/fulfilling_do.go:102","msg":"order fulfilling worker 1: order 1 done"},
{"level":"debug","ts":"2024-03-21T19:30:44.896-0700","caller":"job_manager/manager.go:78","msg":"order fulfilling worker 1: end high priority job (order id: 1)"},

Explicitly declared credentials for dns01goacme

Modifying the configuration slightly renders a slightly different error message, but I think it supports my IMDS theory. If, instead of referencing the credential file, I specify AWS_ACCESS_KEY_ID="MY_VALID_KEY" and AWS_SECRET_ACCESS_KEY="MY_VALID_SECRET_KEY", I get the following log output instead. The error suggests I don't have valid credentials, but invoking aws route53 list-resource-record-sets --hosted-zone-id from the terminal verifies that I do have that IAM permission for these credentials.

{"level":"debug","ts":"2024-03-21T20:39:36.423-0700","caller":"challenges/provisioning.go:25","msg":"added resource for int.my_valid_domain.com to challenge work tracker"},
{"level":"debug","ts":"2024-03-21T20:39:36.769-0700","caller":"challenges/provisioning.go:86","msg":"removed resource for int.my_valid_domain.com from challenge work tracker"},
{"level":"error","ts":"2024-03-21T20:39:36.769-0700","caller":"challenges/solver.go:65","msg":"challenge solver deprovision failed (route53: operation error Route 53: ListResourceRecordSets, https response error StatusCode: 403, Req
uestID: ec0679c7-20c7-45c9-975e-e95b65939770, api error InvalidClientTokenId: The security token included in the request is invalid.)","stacktrace":"legocerthub-backend/pkg/challenges.(*Service).Solve.func1\n\t/home/runner/work/lego
certhub/legocerthub/pkg/challenges/solver.go:65\nlegocerthub-backend/pkg/challenges.(*Service).Solve\n\t/home/runner/work/legocerthub/legocerthub/pkg/challenges/solver.go:71\nlegocerthub-backend/pkg/domain/authorizations.(*Service).
authWorker\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:132\nlegocerthub-backend/pkg/domain/authorizations.(*Service).fulfillAuth\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authori
zations/fulfiller.go:106\nlegocerthub-backend/pkg/domain/authorizations.(*Service).FulfillAuths.func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/authorizations/fulfiller.go:28"},
{"level":"error","ts":"2024-03-21T20:39:36.769-0700","caller":"orders/fulfilling_do.go:101","msg":"order fulfilling worker 2: fulfill auths error: %!w(*fmt.wrapError=&{route53: operation error Route 53: ListResourceRecordSets, https
 response error StatusCode: 403, RequestID: eaaacabe-c917-4a78-abd5-0ae9581388f8, api error InvalidClientTokenId: The security token included in the request is invalid. 0xc000720c60})","stacktrace":"legocerthub-backend/pkg/domain/or
ders.(*orderFulfillJob).Do\n\t/home/runner/work/legocerthub/legocerthub/pkg/domain/orders/fulfilling_do.go:101\nlegocerthub-backend/pkg/datatypes/job_manager.(*Manager[...]).doJob\n\t/home/runner/work/legocerthub/legocerthub/pkg/dat
atypes/job_manager/do.go:22\nlegocerthub-backend/pkg/datatypes/job_manager.NewManager[...].func1\n\t/home/runner/work/legocerthub/legocerthub/pkg/datatypes/job_manager/manager.go:77"},
{"level":"info","ts":"2024-03-21T20:39:36.774-0700","caller":"orders/fulfilling_do.go:102","msg":"order fulfilling worker 2: order 1 done"},
gregtwallace commented 3 months ago

If you are using quotes around the values of your variables, remove them.

I need to add to my to-do list to deal with if people do that since it’s a pretty common format.

If that’s not it, let me know.

sapid commented 3 months ago

:facepalm: Yep, that was it. I didn't look closely enough at config.example.yaml to notice that environment variables don't share the same format as all other arguments.

I'll also make some new screenshots for dns01 acme.sh and dns01 go-acme le-go and make a PR.

sapid commented 3 months ago

I submitted https://github.com/gregtwallace/legocerthub.com-docusaurus/pull/2 for this issue.

gregtwallace commented 3 months ago

The new version is building and the this should work now with or without quotes. Thanks for raising this. The update should avoid what might be a common confusion since the param format often includes quotes.