[Feature]: Platform operator can configure Korifi components to trust a self-signed/internal CA when interacting with the container registry

[tcdowney]

Blockers/Dependencies

No response

Background

Self-hosted container registries may use self-signed certificates or certs that were issued by an internal CA for various reasons (dev envs, air-gapped envs, compliance, etc.). Some of the Korifi components talk directly to the container registry (korifi-api does during package upload and kpack-image-builder does to fetch image metadata) and need to be able to trust these internal registries.

As a Platform Operator I want to be able to configure Korifi with a secret containing a CA to trust for registry interactions So that the Korifi components (korifi-api, kpack-image-builder) can interact with an internal registry

---

Acceptance Criteria

I think we should draw inspiration from how Knative and Cartographer Convention Controller support this:

• Knative sets up a custom transport for the go-containerregistry client they use to talk to the registry. We use the same libraries so could do something similar: https://github.com/knative/serving/blob/492d71c96e7ada509f1cfeae5a8a04f1755fc9ef/pkg/reconciler/revision/resolve.go#L66
• Convention Controller allows the operator to configure an "additionalCACertPaths" config and then merges the contents of this with the system's trusted CAs: https://github.com/vmware-tanzu/cartographer-conventions/blob/97f4f981fbc5b074325a80d4e0008e3238cb80e0/pkg/binding/image_config.go#L171-L190

I think both of these approaches take a Secret and mount it at either a well-known path or provide the path via an env var to the codebase. We should make this Secret optional and operators should be able to use the certs in their regular trust store by default.

GIVEN I have deployed an internal registry that is using a cert signed by an internal CA AND I have configured a Secret with the additional CAs I want to trust WHEN I update the korifi-api and kpack-image-builder Deployments with that secret volume mounted at some path THEN I see that I am able to cf push --no-start an app AND somehow test that the kpack reconciler can read metadata (we may need to use the cert-injection-webhook to get kpack to trust the registry -- if we do this then we can use a regular cf push)

---

Dev Notes

• Kpack does not support self-signed/internal CA configuration out of the box (see: https://github.com/pivotal/kpack/issues/207) and providing support for that is not something in scope for us to take on. Operators can modify the Kpack deployment, use something like the cert-injection-webhook on the kpack Pods, or bring their own build reconciler in these cases.
• Integration testing this may be difficult. We may want to use the cert-injection-webhook approach to get kpack working with it in our test environment.
• Knative docs for similar config: https://knative.dev/docs/serving/tag-resolution/

[akrishna90]

We spent sometime over getting the dockerregistry to deploy locally and serve over tls, unfortunately the changes we were passing to helm weren't working like we expected. We instead switched plans and decided to deploy dockerregistry to a gke cluster, and configure korifi (deployed on local kind) to use it as container registry

Installation instruction for setting up docker-registry with tls on gke

• Claim a GKE cluster

• Add a new DNS entry on cloud for routing connections to docker-registry. In our case we added a docker.beatable-spacesuit.k8s-dev.relint.rocks entry

• Generate CA cert

  
  openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem

- Generate server certs
    - create a `san.cnf` file with contents as below (make sure you update the DNS and CN values)

  [ req ]
  default_bits       = 2048
  distinguished_name = req_distinguished_name
  req_extensions     = req_ext
  prompt = no
  [ req_distinguished_name ]
  countryName                 = US
  stateOrProvinceName         = california
  localityName               = sf
  organizationName           = cf
  commonName                 = docker.beatable-spacesuit.k8s-dev.relint.rocks
  [ req_ext ]
  subjectAltName = @alt_names
  [alt_names]
  DNS.1   = docker.beatable-spacesuit.k8s-dev.relint.rocks
```
- create the CSR
```
openssl req -out sslcert.csr -newkey rsa:2048 -nodes -keyout private.key -config san.cnf
```
- create a `v3.ext` file (we couldn't get the SAN on the server cert, a googling lead us to using this approach. Maybe there is a better way of doing this) - plz update the `subjectAltName` value
```
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints       = CA:TRUE
keyUsage               = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign
subjectAltName         = DNS:docker.beatable-spacesuit.k8s-dev.relint.rocks
```
- create the server cert
```
 openssl x509 -req -days 365000 -set_serial 01    -in sslcert.csr    -out server-cert.pem    -CA ca-cert.pem    -CAkey ca-key.pem -extfile v3.ext
```

• Create a k8s secret for tls

kubectl create secret tls dockerregistry-tls   --cert=server-cert.pem   --key=private.key

• Create a values.yaml file with the below contents. (Make sure you update the hosts value to match yours)

ingress:
  annotations:
    kubernetes.io/ingress.class: "gce"
  className: gce
  enabled: true
  hosts:
    - docker.beatable-spacesuit.k8s-dev.relint.rocks
  tls:
  - hosts:
    - docker.beatable-spacesuit.k8s-dev.relint.rocks
    secretName: dockerregistry-tls

• Install docker-registry using

  helm upgrade --install localregistry twuni/docker-registry  \    
  --set service.type=NodePort,service.nodePort=30050,service.port=30050 \
  --set persistence.enabled=true
  -f "values.yaml"

• Watch the ingress resource status to be updated with a LoadBalancer ip.

• Grab the LoadBalancer IP and update your DNS entry.

• Verify the connection over tls to docker-registry using - On success, you'll get a {} response

  curl https://docker.beatable-spacesuit.k8s-dev.relint.rocks/v2/ --cacert ca-cert.pem