search

cert-manager / cert-manager-olm

Definitions for the cert-manager operator published via Red Hat's Operator Lifecycle Manager (OLM)
Apache License 2.0
14 stars 13 forks source link

Customize the deployment of cert-manager installed via OLM #22

Open hongkailiu opened 3 years ago

hongkailiu commented 3 years ago

Follow up a slack consulting. https://kubernetes.slack.com/archives/C4NV3DWUC/p1619141623473100

If I understand the cert-manager operator correctly, we still need to modify deployment manifests for customizations, such as args of the container such as --dns01-recursive-nameservers="8.8.8.8:53", env vars of the container providing aws credentials, and replicas of the webhook and affinity.

It would be great if the operator can read those customizations from a configMap and reconcile the related manifests.

kfox1111 commented 3 years ago

I think subscriptions can inject env vars into the resulting deployment. maybe it can be configured that way, or maybe that functionality can be added?

MajorFlamingo commented 3 years ago

The customization is also needed to inject custom RootCAs if using a local ACME server, tracked here: https://github.com/jetstack/cert-manager/issues/2332

It looks like a drawback that we cannot configure additional args/mounts.

wallrj commented 3 years ago

@hongkailiu and @MajorFlamingo please read Configuring Operators deployed by OLM and see if any of those can be used to achieve your use-cases.

It might be that we need to update the cert-manager Deployments to allow command line arguments of the cert-manager components to be supplied from environment variables. If so, let me know, and I can either modify the Deployment here in the OLM package or upstream in cert-manager.

Please also comment on the RedHat OCP UI proposal Subscription Config when installing an Operator.

sipe90 commented 2 years ago

It would be great if I could pass the --dns01-recursive-nameservers-only argument to the operator but it doesn't seem to be possible. Also there is no way to set the log level since AFAIK the only way to set it is with the --v argument.

This is the only thing preventing me from using the operator at the moment. As an alternative I've resorted to using the Helm chart instead but the operator would be way more convenient to use in OpenShift.

bdurrow commented 2 years ago

@wallrj, I poked at this and I don't see a way to do it presently. The most straight forward would be to allow for an environment variable that would be used as arguments to the controller. Replacing the existing controller entrypoint with one that called the controller with any additional arguments from that environment variable. Alternatively the controller could be modified to accept some configuration from environment variables but this seems like it would need to touch the code in several different places unless there is a way to do it for using for all options (perhaps using this method).

Requiring that all Issuers use the same values for thing like Recursive DNS server seems like it might be limiting and perhaps changing the CRDs to allow for specifying more of these options in the CR would be a nice target for the future but it seems like the path to that solution would be considerably longer than these other two approaches.

raelga commented 2 years ago

Any update on this matter? :)

wallrj commented 2 years ago

@bdurrow , @raelga and anyone else interested in this, please review

raelga commented 2 years ago

Hey @bdurrow,

Thanks for the updated documentation.

For our use case, we ended developing a new operator using the OperatorSDK Helm framework. It's actually a kind of meta-operator, when the operator image is built, it fetches the helm chart and stores it into the container image.

When deployed into a cluster, it allows every single configuration available in the helm chart using an OperatorConfig instance manifest to define the values.yaml. With that configuration, it renders, deploys and reconciles all the resources from the helm chart.

The code is available here:

https://github.com/3scale-ops/cert-manager-helm-operator

Images are built on GH actions:

https://github.com/3scale-ops/cert-manager-helm-operator/actions

And the images are pushed to quay.io:

A few things about the implementation:

https://github.com/3scale-ops/cert-manager-helm-operator/runs/6691411489?check_suite_focus=true Screenshot 2022-06-02 at 18 15 09

https://github.com/3scale-ops/cert-manager-helm-operator/blob/main/Makefile#L261-L276

An example on how a release looks like is this PR https://github.com/3scale-ops/cert-manager-helm-operator/pull/3/files. Everything in the bundle path is generated from the yamls in config. And most part of those config yamls are also generated from the information available on the helm charts. We added some changes like missing rbac permissions, unauthenticated metrics and enforcing inject-apiserver-ca: true.

apiVersion: operatorconfigs.operator.cert-manager.io/v1alpha1
kind: OperatorConfig
metadata:
  name: cluster
spec:
  prometheus:
    enabled: true
    servicemonitor:
      enabled: true
      targetPort: 9402
      path: /metrics
      interval: 30s
  podDnsConfig:
    nameservers:
      - 1.1.1.1
      - 8.8.8.8
  resources:
   requests:
     cpu: 10m
     memory: 96Mi
   limits:
     cpu: 100m
     memory: 256Mi

Screenshot 2022-06-02 at 18 42 30

Screenshot 2022-06-02 at 18 43 06

Screenshot 2022-06-02 at 18 45 45

Screenshot 2022-06-02 at 18 43 19

QuingKhaos commented 2 years ago

@wallrj this operator needs a extra CertManager CRD which enables to install cert-manager driven via the CRD. IMHO the operator should never install the software it manages directly as a result from the subscription, but provide a CRD to install an instance of the software.

A CertManager CRD can expose all configuration options cert-manager and the components provided, but is still opinionated in the exposed configurations. Example would be the ArgoCD operator. The OLM Subscription config is solely to configure the operator-controller, not the operands an operator deploys.

In the end, this miss is a reason, why the new cert-manager Operator for OpenShift is a separate implementation (providing a CertManager CRD) and not a fork of the upstream operator, like most Red Hat provided operators are.

hongkailiu commented 1 year ago

Thanks for all the input and effort on the topic.

Some update from my side:

The only customization is the webhook (replicas and affinity) which is not a blocker (still nice to have though) for us.

We will start to adopt cert-manager-operator after it is GA (currently still in TechPreview).

absynth76 commented 1 month ago

Just a fyi, but redhat cert-manager operator (openshift) adopted the CertManager CR solution described above https://github.com/cert-manager/cert-manager-olm/issues/22#issuecomment-1214169292