configuring self signed or custom SSL certs for deploykf plugin

mnasty commented 3 months ago

This is a documentation enhancement request asking to add the following information into the official deployKF docs to help other users. The content is written below, inline using markdown as I could not find a way to contribute to the docs directly and this information is not otherwise available to my knowledge.

I wasted alot of time figuring this out and getting it working. It should be written somewhere.

Configuring DeployKF Plugin on ArgoCD for Self Signed or Custom SSL Certificates

This Guide is for DevOps admins who may be deploying in an On Prem datacenter or potentially those with custom SSL configurations in a public cloud VPC

The deployKF plugin downloads dependencies using a sidecar container that runs inside the argocd-repo-server pod. Both the argocd server and sidecar container need to have SSL certs added into their truststore chains to properly authenticate with a self signed/custom new certificate when executing the sync process with argocd. This method has been tested using the argocd sync script.

1.) Store downloaded pem certs as configmap

Using the below command we can load the contents of the certificate we need to add into Kubernetes in a persistent way:

kubectl -n argocd create configmap root-chain --from-file=root-chain.pem

2.) Patch the argocd repo server deployment to include the cert in all it's container images as a volume mount into the container level default truststore (/etc/ssl)

Fill in the yaml spec below to match your configmap name/key and set desired mount paths/attributes:

spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - name: root-tls
              mountPath: /etc/ssl/certs/root-chain.pem
              subPath: root-chain.pem
        - name: deploykf-plugin
          volumeMounts:
            - name: root-tls
              mountPath: /etc/ssl/certs/root-chain.pem
              subPath: root-chain.pem
      volumes:
        - name: root-tls
          configMap:
            name: root-chain
            items:
              - key: root-chain.pem
                path: root-chain.pem

Convert to JSON, minify and use kubectl patch to make the deployment patch persist through the lifecycle of the pods ensuring your SSL cert is always in the chain when a resource needs to be synced

kubectl -n argocd patch deployment argocd-repo-server -p '{"spec":{"template":{"spec":{"containers":[{"name":"argocd-repo-server","volumeMounts":[{"name":"root-tls","mountPath":"/etc/ssl/certs/root-chain.pem","subPath":"root-chain.pem"}]},{"name":"deploykf-plugin","volumeMounts":[{"name":"root-tls","mountPath":"/etc/ssl/certs/root-chain.pem","subPath":"root-chain.pem"}]}],"volumes":[{"name":"root-tls","configMap":{"name":"root-chain","items":[{"key":"root-chain.pem","path":"root-chain.pem"}]}}]}}}}'

Validate by using curl from within the deploykf plugin sidecar container to reach github or custom repo/resource. Default ssl store is /etc/ssl in both containers but only the plugin sidecar has curl installed. Sync your apps as normal after this change.

Proxy variables can be added to all containers in the repo server pod using the following syntax if needed, though be sure to set this BEFORE applying the app of apps yaml file to prevent sync inconsistencies:

kubectl -n argocd set env deployment/argocd-repo-server HTTP_PROXY=http://10.0.0.0:8080 HTTPS_PROXY=http://10.0.0.0:8080 NO_PROXY=argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,mydomain.com

mnasty commented 3 months ago

I just realized the docs issue link took me to a completely different repository where the website can be updated... would love some help getting this typed up and added.

thesuperzapper commented 3 months ago

@mnasty I assume this is for the case that you are hosting your own internal git and/or helm repos with self signed certificates.

I have a few questions:

What were you doing when you found this problem?
What things in your environment use a self signed certificate?
I assume you don't want to just disable SSL verification?

mnasty commented 3 months ago

So this is actually for configuring self signed certificates to be able to download the resources for deployKF in an On Prem datacenter but may have other potential applications as I noted above, and you so aptly pointed out with your assumption, given that on prem as a concept is sunsetting as public cloud takes over.

To be more specific, certs for passing through a proxy server before reaching the internet. If one thing in the chain is not trusted, nothing is.

Your questions:

What were you doing when you found this problem?

Trying to sync the app of apps with the provided sync script and getting ssl errors x509: certificate signed by unknown authority unable to get local issuer certificate

What things in your environment use a self signed certificate?

The internal corporate network uses them as the company is not interested in paying a certificate authority

I assume you don't want to just disable SSL verification?

This is a gaping security hole in a high-compliance production environment and would require a similar amount of effort to just adding the certificates into the trust chain to begin with

deployKF / website