ACM service controller

[mdykes-gw]

New ACK Service Controller

Support for ACM

List of API resources

List the API resources in order of importance to you:

1) Certificate

[jaypipes]

@mdykes-gw can you elaborate on how you'd envision an ACK service controller for ACM working? I suppose the only resource in the ACM API is the Certificate resource. However, instead of a Create call, there is only an ImportCertificate API call. There are API calls like ExportCertificate that don't make sense in a Kubernetes resource model world (there is no state to reconcile...).

Were you thinking of implementing your own kind of certificate renewal using an ACK service controller for ACM and some other script or Kubernetes operator?

[Vrtak-CZ]

From my point of view it can initiate creation of Certificate so this certificate can be used with AWS LoadBalancer Controller. The problem is that this can be tricky because it will probably also need access to Route 53 resource for validation and it will work only for domains hosted in Route 53.

[Comradin]

@Vrtak-CZ But isn't this basically the exact use-case for many EKS users?

We have EKS, Route53, and the Loadbalancer Controller and no automatic certificate issuer for dynamic scopes. We had to fall back to pre-defined certificates using tools like pulumi or terraform. This breaks the envisioned workflow of just creating a development instance with a valid certificate chain

Route53 in our setups have an authoritative domain like dev.example.com

The developers should be enabled to deploy an ingress host for my-service.my-dev-namespace.dev.example.com and everything works in an automated way.

[ack-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

[Vrtak-CZ]

/remove-lifecycle stale

[Pitta]

I came here looking for exactly this workflow. Would prefer to not use a wildcard cert. Use wise, being able to include an annotation to create the cert in acm would be useful.

metadata:
  annotations:
    aws.acm.kubernetes.io/create: true

The aws-load-balancer-controller should wait till there is a valid cert if using the certificate lookup method. So if this listened for ingress objects and acted on that annotation, it would jive with the workflow of the aws-load-balancer-controller.

---

Or just use a similar Certificate kind used by CertManager and grant the controller to specified Route53 hosted zones and Cert Manager for creating new certs.

[RedbackThomson]

@Pitta I think these are good suggestions, I'm just not sure if ACK is the place for them.

AWS already supports using ACM for private certificates through the cert-manager issuer - https://github.com/cert-manager/aws-privateca-issuer/

Perhaps some integration between that issuer and the aws-load-balancer-controller would be more appropriate? It is not within the scope of ACK to interact with Ingress resources - only our own custom resources.

[sbkg0002]

Why the forced push on the private ca?

If cloudformation or the cli can do it, this should also be able right?

[Pitta]

I've been working on a simple controller to handle this based on kopf at work. Hoping I can convince the org to start making some stuff open source, or at least let me fork the code properly so I can make it public outside the org.

That said, the logic is pretty simple. I haven't done the service account yet, but the operator worked in my local testing.

[sbkg0002]

Thanks for sharing @Pitta !looking forward.

[RedbackThomson]

@sbkg0002 Yes CloudFormation and the CLI are both able to create certs, and an ACM ACK controller would be able too as well. However, we don't have plans to extend ACK out any further than the control plane of ACM, for example attaching certificates within a Kubernetes environment.

I think there are other good tools (like cert-manager) that handle the K8s part of it once the certificate has been created by an ACK custom resource.

[Pitta]

If cert-manager did what I was asking for in a way that was clear that others have done, I'm all for it. Nothing I've found in my travels suggests that anything out of the box will do ACM public cert management.

[vijtrip2]

/lifecycle frozen

[olemarkus]

The ACM + route53 validation + Load balancer controller use case is definitely very interesting to us. Not having this functionality prohibits us from using ACM and ALBs.

[ohookins]

I also have this use case and the most recently linked issue is mine. Like another previous commenter I'm a bit mystified why there seems to be more support in the ecosystem for ACMPCA rather than public ACM certificates - so many use cases for dynamic environments require a load balancer, publicly-verifiable certificates and matching DNS.

To be honest I don't know why the ALB, ACM and Route53 services aren't more tightly integrated to make this a one-click operation, but that's a separate feature request ;)

[erhudy]

I too am interested in this. I think it's worth distinguishing between the IssueCertificate and RequestCertificate API calls, because they do different things. The cert-manager plugin for ACMPCA seems to only do IssueCertificate, which is an ACMPCA-exclusive call - this requires you to generate a key and CSR and submit it to the PCA, and then you get a certificate back. It does not seem to support RequestCertificate, which can be used both for ACM public certs and also for getting certs from ACMPCA that you want to use in other managed services like ALB.

This RequestCertificate gap is what I would like to see solved; we use ACMPCA now but have to tell people that they need to provision certs themselves through Terraform/API/console and get the ARN, then use that ARN in the annotation for the LB controller. It's not wretched, but the ideal workflow would be annotating a Service or Ingress in a way that tells the LB controller to deal with requesting the certificate itself and figuring out the SANs based on the NLB hostname annotation or Ingress hosts, respectively.

[matheushent]

If it matters, there still are people interested in this, e.g. me!

[migueleliasweb]

It's 2023 now.... 1 year and 2 months of this thread. This shouldn't be taking this long.

It's quite sad that some are still trying to untangle all the details of an advanced implementation (ACM with Private CAs + LB attachment) but the basic functionality of just creating a simple public ceritificate using DNS challenge is not even supported yet.

Can we just agree of having a simple support for creating public certs first then we can iterate over that to then add support for more advanced cases?

Happy 2023 🎉

[jaypipes]

but the basic functionality of just creating a simple public ceritificate using DNS challenge is not even supported yet.

If only it were actually a simple thing. :)

Can we just agree of having a simple support for creating public certs first then we can iterate over that to then add support for more advanced cases?

After digging into the ACM APIs, I think we could support Certificate resources with the RequestCertificate API as the Create operation, DeleteCertificate as the Delete operation, DescribeCertificate API as the ReadOne operation and UpdateCertificateOptions as the only Update operation.

We would need to add a caveat, though:

If no certificateAuthorityARN is specified, the Certificate will be a public one. In this case, validationMethod will be hard-coded to DNS (instead of Email) and the number of domainValidationOptions will be limited to 5 (because more than 5 mean that email verification is needed, and that's not an automateable thing.

We could handle the ImportCertificate path at a later date.

@migueleliasweb would that meet your minimum use case needs?

[migueleliasweb]

Thanks for jumping in, @jaypipes . I think you've nailed the problem.

I did basically the same digging as you did and I didn't think that was overly complicated. I'm sure there's a lot of people here in the AWS team (and in the broader community) that are on top of the APIs, so my rationale was that the main problem here is having a plan in the first place and not really implementing it. That's basically why I thought it was sad this thread is taking over a year to output any sort of outcome.

Just to be clear, the usecase you mentioned won't fully solve my usecase as I would still like to have some kind of way to attach this to a LB (I was thinking there could be something like a CertificateAttachment kind we could use to attach a given cert to a LB asynchronously).

Have said that, this is lightyears better than nothing! I will take it!

Taking smaller steps like this will give all the involved parties more confidence they're heading the right direction. This (from my point of view) is far better than trying to foresee 2..3..5..10 steps ahead and ended up overcomplicating something that could have given value to the community much sooner.

[jaypipes]

@migueleliasweb @olemarkus @matheushent @ohookins @erhudy OK, so I created the controller today and got the Certificate resource generated properly. Tomorrow I will work on e2e tests of the Certificate resource and then if all goes to plan, cut an initial v0.0.1 release of the controller by Friday.

[olemarkus]

This looks good!

[matheushent]

@jaypipes This is great news, thank you for the effort!

[migueleliasweb]

What a legend! @jaypipes

ACK > TF

There, I said it! 🤣

[jaypipes]

What a legend! @jaypipes

ACK > TF

There, I said it! rofl

LOL, thank you @migueleliasweb, but Terraform is an excellent tool. We try not to compare to it, because TF is trying to be a lot more than ACK is, and ACK is Kubernetes-centric while TF definitely is not.

But, it's nice to hear that you like ACK!

[migueleliasweb]

We try not to compare to it, because TF is trying to be a lot more than ACK is, and ACK is Kubernetes-centric while TF definitely is not.

Come on mate (@jaypipes ), give yourself some credit! I'm just here watching you smash those PRs in the last couple of weeks!

And yeah, I totally understand the difference. Part of my role on my new gig is to move torwards (what I call) Kubernetes-native platform, instead of a "somewhat-cloud-native" platform (which is what used to be with lots of bolt-on scrips everywhere :sweat_smile: ).

Once again, thanks for the help. I'm very much looking forward to use the ACM controller :wink: .

[jaypipes]

Thanks for kind words @migueleliasweb :)

@migueleliasweb @olemarkus @matheushent @ohookins @erhudy you can install and use the ACM controller for ACK now. We released v0.0.1 of the acm-controller two days ago.

You can install it using the acm-chart Helm chart.

Feedback and criticism very much welcomed! -jay

[matheushent]

@jaypipes Thank you very much for you work on this, I (and possibly other folks) appreciate it very much.

Now, about the next steps, what are the plans to attach the certificate created by the controller to a load balancer? I believe this is the main use case for pretty much everyone interested on this, so, I'm willing to get it going. I'm able to help if needed as well.

[Pitta]

Doesn't the aws load balancer controller do certificate discovery?

[matheushent]

Doesn't the aws load balancer controller do certificate discovery?

yes, I missed it. Thank you very much. Docs here for who's interested.

[migueleliasweb]

Now, about the next steps, what are the plans to attach the certificate created by the controller to a load balancer?

Hi @matheushent , I assume you're not using the AWS Load Balancer Controller.

Otherwise you could attach certs to a ALB/NLB easily with it: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/ingress/annotations/#certificate-arn

It also has certificate discovery: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/ingress/cert_discovery/

[tavlima]

Hi, @jaypipes.

My use case is a bit different: I already have cert-manager generating certificates for our cluster, using our private CA, but since we use the AWS LBC with cert discovery, it would be great if we could import those certificates in ACM.

Our current workaround is to use Terraform to generate another certificate (for the same DNS as the Ingress) and import that into ACM, for the cert discovery to work.

Could that be accommodated in this ACM service controller?

[jaypipes]

@tavlima I can work on an ImportCertificate workflow for ACM, sure. Unfortunately I won't be able to get to this for probably another couple weeks, though.

[tavlima]

That would be awesome, @jaypipes!

FWIW, this is the cert-manager feature we use. Perhaps it would make sense for this service controller to export any certificate generated by a given cluster issuer?

A couple of caveats, though, is reconciling these certificates (as Ingress resources come and go), as well as the case of multiple EKS clusters in the same account. But I guess you already thought of all these. 😅

[calvinbui]

great work @jaypipes

are there plans to automate validation through route 53 or by a helper utility?

[jaypipes]

great work @jaypipes

are there plans to automate validation through route 53 or by a helper utility?

Hi Calvin! At this time, no, I don't have plans to do this. As we complete the route53 controller, however, we can certainly look into this. Would it be possible for you to create a separate GH issue for the route53 validation behaviour feature request, since this GH issue is for the ACM service controller generally?

[akamac]

Our use case is to import certs, issued by cert-manager, to ACM. It would be great if the operator can do this.

[FernandoMiguel]

Our use case is to import certs, issued by cert-manager, to ACM. It would be great if the operator can do this.

@akamac ACM certs are free, why not issue new ones instead?

[akamac]

Our use case is to import certs, issued by cert-manager, to ACM. It would be great if the operator can do this.

@akamac ACM certs are free, why not issue new ones instead?

Because AWS no longer issues certificates for ru zone, while Let's Encrypt does.

[mahadh02]

@tavlima I can work on an ImportCertificate workflow for ACM, sure. Unfortunately I won't be able to get to this for probably another couple weeks, though.

@tavlima , Wondering if you have any updates on the ImportCertificate workflow? TIA

[jaypipes]

@tavlima I can work on an ImportCertificate workflow for ACM, sure. Unfortunately I won't be able to get to this for probably another couple weeks, though.

@tavlima , Wondering if you have any updates on the ImportCertificate workflow? TIA

@mahadh02 it was me who was working on this :) And I have not had the time or resources to get to it unfortunately. I may have some time towards the end of May to tackle this, however.

[mahadh02]

@mahadh02 it was me who was working on this :) And I have not had the time or resources to get to it unfortunately. I may have some time towards the end of May to tackle this, however

@jaypipes , my apologies for incorrectly quoting.. this feature will be very beneficial and looking forward for its availability.

[kstevensonnv]

@jaypipes Echoing what the initial comments said.

We're moving to EKS and I was looking to use this controller to create a certificate and complete validation.

If this controller can satisfy that, we could use AWS Load Balancer Controller to act on the ingress resource with simply a host set and it would all just work nicely (in theory).

Now I'm looking at implementing what we currently have outside EKS which feels like a step back.

[FuriouZ07]

DNS validation via R53 record set is the last missing piece for creating a service with ELB and certificate with r53 alias through Kubernetes manifests...unfortunately, as long as this is missing, the acm-controller is not usable efficiently.

[Gianluca755]

@FuriouZ07 shouldn't that be done with external-DNS + acm-controller? General question: does the acm-controller work with annotations on an ingress resource or it needs a crd?

[FuriouZ07]

@FuriouZ07 shouldn't that be done with external-DNS + acm-controller? General question: does the acm-controller work with annotations on an ingress resource or it needs a crd?

Yeah, external-dns could create a R53 record. But it needs to query the cname name&value from the ACM certificate and then create a record with these data. As far as I know, external-dns does not support such a use case to automate the creation&verification completely.

I've used the CRD for creating the certificate and did not try to use annotations.

[indrekj]

I'm also not finding a way to automate the R53 DNS validation. Everything else worked well, but the validation is a blocker for us as well.

[tavlima]

@tavlima I can work on an ImportCertificate workflow for ACM, sure. Unfortunately I won't be able to get to this for probably another couple weeks, though.

@tavlima , Wondering if you have any updates on the ImportCertificate workflow? TIA

@mahadh02 it was me who was working on this :) And I have not had the time or resources to get to it unfortunately. I may have some time towards the end of May to tackle this, however.

Hey, @jaypipes. Happy new year. 😄 Any updates on this?

[john-r-swyftx]

@jaypipes I can create a validated cert by doing the steps in https://github.com/aws-controllers-k8s/community/issues/1904#issuecomment-1909358022

But there is the missing part of getting the resource record info to create the correct RecordSet resource. Can that info be put onto the Certificate status like aws acm describe-certificate --certificate-arn <arn> does?

I could work with that for now

[a-hilaly]

Can that info be put onto the Certificate status like aws acm describe-certificate --certificate-arn does?

@john-r-swyftx Resource recordsets are now shown in the Certificate status - checkout acm-controller v0.0.12/13 https://github.com/aws-controllers-k8s/acm-controller/pull/29/files