file permission errors in the certificates init container

blancharda commented 1 month ago

Environment

Device and OS: RHEL VMs on Nutanix provisioned infra App/package versions: v17.0.1 Kubernetes distro being used: RKE2

Steps to reproduce

deploy gitlab
observe logs in the certificates init container on the webservice pod

Expected result

No errors.

Actual Result

cp: cannot create regular file '/usr/share/pki/ca-trust-source/anchors/ca.crt': Permission denied
cp: cannot create regular file '/usr/share/pki/ca-trust-source/anchors/cfssl_ca': Permission denied
Copied custom certificates to RHEL-compatible directory
p11-kit: couldn't create file: /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt: Unknown error 13
p11-kit: couldn't create file: /etc/pki/ca-trust/extracted/java/cacerts: Unknown error 13
p11-kit: couldn't create file: /etc/pki/ca-trust/extracted/edk2/cacerts.bin: Unknown error 13
Updated CA trust
Copied bundles into /etc/ssl/certs

Severity/Priority

low -> mid I'm not sure the extent of the impact here given that most environments seem to be functional, but I could see this causing some odd behavior down the line if not addressed.

Additional Context

This may be related to Pepr's user/group mutation on the init container
This may be an issue specific to the registry1 image
The OS is FIPS enabled? (not sure this is related.. but might be relevant)

blancharda commented 1 month ago

fwiw -- adjusting the init container's runtime user to match the registry1 image user with the following override did not resolve the behavior:

  - name: gitlab
    repository: ghcr.io/defenseunicorns/packages/uds/gitlab
    ref: 17.0.1-uds.1-registry1
    overrides:
      gitlab:
        gitlab:
          values:
            - path: gitlab.webservice.init.containerSecurityContext
              value:
                runAsNonRoot: true
                runAsUser: 65534

(Note that the GL chart doesn't appear to expose the security context for each individual init container.. Applying the above broke the dependencies initcontainer which had to be manually adjusted in-cluster in order to test a "successful" deployment)

blancharda commented 1 month ago

Dropping some links to various pieces of the helm chart here as well because they were a bit of pain to track down:

webservice deployment
certificates container helper template
init container security context template
webservice values file (the docs list the key in the wrong place 🙃)

oates commented 1 month ago

We could check to see if we see this in the staging env @defenseunicorns/swf

Racer159 commented 3 weeks ago

Observed this in staging - it currently only affects the registry1 image - long term we would want to have a way (i.e. through trustmanager) to ensure that the certs are what we want them to be with no other certs in the truststore.

Racer159 commented 3 weeks ago

Of note adding certs to GitLab works for webservice but not registry and the GitLab chart does not expose extra volume mounts in the registry...

Racer159 commented 3 weeks ago

Blocking this on https://github.com/defenseunicorns/uds-core/issues/464 for now - we could implement our own path / workarounds but it would be best to integrate with uds-core

defenseunicorns / uds-package-gitlab