search

cert-manager / aws-privateca-issuer

Addon for cert-manager that issues certificates using AWS ACM PCA.
Apache License 2.0
184 stars 77 forks source link

[Bug]: Issuer is not ready and certificate creation fails #292

Closed hiroo-kitamura closed 11 months ago

hiroo-kitamura commented 11 months ago

Describe the expected outcome

I created a certificate included in examples according to the README on OpenShift, but the creation fails.

I couldn't find any clear restrictions in the README, but is it possible to run this project on systems other than Amazon EKS, such as OpenShift or vanilla kubernetes? If it is possible to make it work, how can I link the IAM Role listed in Configuration to the Service Account?

Describe the actual outcome

The following error occurs when creating a certificate resource.

  Warning  Failed     60s   cert-manager-certificates-issuing          The certificate request has failed to complete and will be retried: issuer is not ready

Steps to reproduce

Relevant log output

* secret

$ oc describe secret/example
Name:         example
Namespace:    cert-manager
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
AWS_ACCESS_KEY_ID:      21 bytes
AWS_SECRET_ACCESS_KEY:  41 bytes

Version

v1.2.5

Have you tried the following?

Category

Build Issues

Severity

Severity 1

Hamidhasan commented 11 months ago

Thank you for raising this issue with the AWS Private CA Issuer plugin. We will review your submission and respond back to you here as soon as possible.

Hamidhasan commented 11 months ago

It should be possible to get the plugin to work by simply using the access key / secret; EKS is not mandatory.

Are you sure your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values are Base64 Encoded? They will not work otherwise.

Hamidhasan commented 11 months ago

I tried for a few times to reproduce the issue -- ultimately, was not able to. A few times I did actually reproduce the situation you had, but in my case, I was trying it with a generated access key id / secret key and I think my credentials were expired when it tried to issue the cert. I also tried when I had not base64 encoded the creds, and got the issue you experienced. Also, I did it a few times where the secret name mismatched (it does not inform you if the secret fails to parse, or is expired, or is invalid, etc., when creating/describing the AWSPCAIssuer, it will just fail the cert issuance). It took me a few tries admittedly, but I did get it to work using the access key / secret key method.

When using an access key, it will work only via the a permanent one, i.e. one created via https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey, instead of generated temporary credentials. This made it much easier, and worked for me on the first try.

I tested a little bit more, and edited this comment. I can confirm that I can reproduce the issue only when using temporary credentials, but when using a permanent one, it will work correctly. Can you comment on how you generated the access key and secret key? Did you do it via temp credentials (usually there is an AWS_SESSION_TOKEN with this method), or did you do it with an Access Key created in IAM (i.e. from the above link)?

hiroo-kitamura commented 11 months ago

Thank you for the answer.

Can you comment on how you generated the access key and secret key?

I encoded these with the following commands. I also performed a decoding check.

$ echo <AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY> | base64
<Encoded ID or KEY>
$ echo <Encoded ID or KEY> | base64 -d
<AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY

Did you do it via temp credentials (usually there is an AWS_SESSION_TOKEN with this method), or did you do it with an Access Key created in IAM (i.e. from the above link)?

I did it with an Access Key created in IAM. More precisely, I created it with CLI rather than console, but I don't think there is essentially a difference.

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey_CLIAPI

Hamidhasan commented 11 months ago

Thanks for the reply. That method should be supported, and I verified on a small setup that it worked last night.

Couple more questions:

  1. What is the version of cert-manager vended by OpenShift?
  2. What version of the aws-privateca-issuer plugin are you using?
  3. What is the IAM policy associated with the Access Key you are using?

Potentially OpenShift is vending an older version where things don't quite work correctly; however, this capability should be supported by older versions. Alternatively, the IAM policy may not be permissive enough.

hiroo-kitamura commented 11 months ago

Thank you for the answer.

1.What is the version of cert-manager vended by OpenShift?

cert-manager Operator for Red Hat OpenShift v1.11.4

2.What version of the aws-privateca-issuer plugin are you using?

v1.2.5

3.What is the IAM policy associated with the Access Key you are using?

The following policy is assigned to the IAM user. I have configured settings what is listed in Configuration.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "awspcaissuer",
      "Action": [
        "acm-pca:DescribeCertificateAuthority",
        "acm-pca:GetCertificate",
        "acm-pca:IssueCertificate"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:acm-pca:<region>:<account_id>:certificate-authority/<resource_id>"
    }
  ]
}
hiroo-kitamura commented 11 months ago

I've done some additional verification and will share it. I've checked to see if there was any problem with the connectivity from aws-privateca-issuer's Pod to AWS environment. After checking, there seemed to be no problem.

$ oc -n cert-manager get pod
NAME                                                  READY   STATUS    RESTARTS   AGE
aws-privateca-issuer-v1-1696905727-5c5dd95887-tpgpr   1/1     Running   0          3h38m
cert-manager-5fb76fbd98-wl5rp                         1/1     Running   0          3h38m
cert-manager-cainjector-fc95dff7b-hsptf               1/1     Running   0          3h38m
cert-manager-webhook-5f677fbff5-b9m6z                 1/1     Running   0          3h38m

$ oc -n cert-manager debug aws-privateca-issuer-v1-1696905727-5c5dd95887-tpgpr -t --image=amazon/aws-cli -- sh
Starting pod/aws-privateca-issuer-v1-1696905727-5c5dd95887-tpgpr-debug, command was: /manager --leader-elect -disable-approved-check
Pod IP: <mask>
If you don't see a command prompt, try pressing enter.
sh-4.2$ export AWS_ACCESS_KEY_ID=<mask>
sh-4.2$ export AWS_SECRET_ACCESS_KEY=<mask>
sh-4.2$ cd /tmp/
sh-4.2$ mkdir .aws
sh-4.2$ cat <<EOF >.aws/credentials
> [default]
> aws_access_key_id = ${AWS_ACCESS_KEY_ID}
> aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}
> EOF
sh-4.2$ aws iam get-policy --policy-arn "<mask>"
{
    "Policy": {
        "Arn": "<mask>",
...snip...
    }
}
hiroo-kitamura commented 11 months ago

I tried various ways and was able to successfully deploy the certificate resource as below.

$ oc get certificate
NAME            READY   SECRET          AGE
rsa-cert-2048   True    rsa-cert-2048   22s

When it was successful, I intentionally registered AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as a secret without Base64 encoding. I don't know why this way was successful, but does such a case exist? If so, I would appreciate it if you could update the documentation.

anufreichik1 commented 11 months ago

Thank you for update on this issue with the AWS Private CA Issuer plugin. We will review your request on documentation update and respond back.

Hamidhasan commented 11 months ago

Thanks for trying again Hiroo, and apologies for the confusion. Happy that it worked for you!

We investigated more after your comment, and realized that actually, both are allowed. It's the Kubernetes Secret type that supports both. Basically, the Secret type can take either plaintext String data, or base64 data. This article describes has more information.

From the article, here are a few examples. Our issuer works the same way, you can either use plaintext AccessKeyId/Secret in secretData, or base64 encoded AccessKeyId/Secret in data. For both, the keys would be AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

With stringData:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
stringData:
  AWS_ACCESS_KEY_ID: testAccessKeyId
  AWS_SECRET_ACCESS_KEY: testSecretAccessKey

With data: (base64)

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
data:
  AWS_ACCESS_KEY_ID: dGVzdEFjY2Vzc0tleUlk
  AWS_SECRET_ACCESS_KEY: dGVzdFNlY3JldEFjY2Vzc0tleQ==

Also make sure to read the notes on base64 encoding and newlines, there can be issues if trying to copy paste from one tool to another with base64 encoding and trailing newlines.