kwctl does not honor trusted certificate authorities on the system

Martin-Weiss commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

kwctl pull against on-premise registry with self-signed certificate / certificate authority fails with:

kwctl pull registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0

2022-11-22T15:54:48.099449Z  WARN rustls::conn: Sending fatal alert BadCertificate    
Error: the policy registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0 could not be downloaded due to error: error sending request for url (https://registry01.suse:5000/v2/): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

Expected Behavior

The tool should trust the CA of the systems ca-bundle.pem.

A cli parameter to specify the trusted CA bundle should be available.

Steps To Reproduce

Use on premise registry for policies and try to use pull / push with kwctl.

Environment

- OS: SLES 15 SP4
- Architecture: x86_64

Anything else?

No response

Martin-Weiss commented 1 year ago

Yes - there seems to be a way to use the sources.yaml with kwctl but here I can not specify the certificate of the CA in a way that works:

source_authorities:
 "registry01.suse:5000":
 - type: Data
   data: |
    -----BEGIN CERTIFICATE-----
    MIIDhzCCAm+gAwIBAgIUVS8hv/sK6r15ZjVf/Zz/AQdoqjAwDQYJKoZIhvcNAQEL
    BQAwUzELMAkGA1UEBhMCREUxGzAZBgNVBAgMEkJhZGVuLVd1ZXJ0dGVtYmVyZzEN
    MAsGA1UECgwEU1VTRTEYMBYGA1UEAwwPcmVnaXN0cnkwMS5zdXNlMB4XDTIxMDMy
    NDE0MTMzM1oXDTMxMDEzMTE0MTMzM1owUzELMAkGA1UEBhMCREUxGzAZBgNVBAgM
    EkJhZGVuLVd1ZXJ0dGVtYmVyZzENMAsGA1UECgwEU1VTRTEYMBYGA1UEAwwPcmVn
    aXN0cnkwMS5zdXNlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqpHI
    1l8C83ON0MGkKA/s+bfJnFjBrHdeHoGbxnxqDk63cXLWIejI5QmUNGIKlGUpLnwl
    It6QDtK5DRXfxR70MRDWu8p7r2yc5Vwrj/z0GqdTFFxsu+o832fukJQLAVTt2O1k
    bH7PDzxsz9yewTNS5dRKmk27+kRjqIvudvUo2+LC/DmMpZiaXDXf60tgdHGhv38n
    ZCqj6S2I/zh1pk2FqCz1ShQRJ8zAist/RLlO77vU30o1pHaIbEjKBj6I6O7SX6SO
    NOBdDQ7zUEKmtmADVTQaDNBw1aJBpUtOYKAWlZyrwzmgXLFXPYcJLq7HEi9RdF2O
    9P7SN/toZW5vsyXqxwIDAQABo1MwUTAdBgNVHQ4EFgQUlqMA2BU9MN9HRH7ebcqj
    VjGxt6owHwYDVR0jBBgwFoAUlqMA2BU9MN9HRH7ebcqjVjGxt6owDwYDVR0TAQH/
    BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAE9/UQ1qSi3932Lq6OJ48NkSa12V0
    ezxE7fgDvm4wWK5J83yToNMp6K0uO3PfKWko7+ancgT3n2+gw5DO4/9YUplkszzo
    KISVWnh+ILDHCjhzKgJNJP/zpa5bWkrvTa+WYcXqxMndYsokwd9sI/gYUtCMMngp
    4VBOPlephUpUriB+sT6e/ODszXawLXNpM1oww2k0DOoxrF+KOMldVuqo/LXHOElR
    amXoZIEHPRiXltNGg6NdyeSgMX0qugYyNpHRnwvwVqa9+76cOFfrQqRqq8ueeZY3
    T7s2fY/AJ5Eg66wfhxSm8rNf31TWCz8eaqlTm5kd5Fzf2vwZp84pnDmLRg==
    -----END CERTIFICATE-----

Results in

kwctl pull --sources-path kwctl-sources2.yaml registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0
2022-11-22T16:23:02.422983Z  WARN rustls::conn: Sending fatal alert DecodeError   
Error: the policy registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0 could not be downloaded due to error: error sending request for url (https://registry01.suse:5000/v2/): error trying to connect: invalid peer certificate encoding

flavio commented 1 year ago

Yes - there seems to be a way to use the sources.yaml with kwctl but here I can not specify the certificate of the CA in a way that works: [...] Results in

kwctl pull --sources-path kwctl-sources2.yaml registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0
2022-11-22T16:23:02.422983Z  WARN rustls::conn: Sending fatal alert DecodeError   
Error: the policy registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0 could not be downloaded due to error: error sending request for url (https://registry01.suse:5000/v2/): error trying to connect: invalid peer certificate encoding

This turned out to be caused by the usage of the wildcard certificate like *.suse. Using a wildcard certificate like *.suse.local works fine. It looks this behavior is caused by rustls, I've reached out to upstream and I talking about this issue.

The issue reported when the issue got created is valid. We have to look into that. It looks like the reqwest crate, which is being used by oci-distribution, has a feature that enables the usage of system certificates:

rustls-tls-webpki-roots: Enables TLS functionality provided by rustls, while using root certificates from the webpki-roots crate.

This is something worth to be investigated

viccuad commented 1 year ago

Opened https://github.com/kubewarden/policy-fetcher/issues/126 also related to this, they could be merged.

viccuad commented 1 year ago

Working on this, will edit this post.

Patched kwctl -> policy-evaluator -> policy-fetcher -> oci-distribution with reqwests rustls-native-certs feature and rustls-native-certs as new dependencies as needed.

To test, deployed Harbor via Helm on a minikube cluster:

minikube addons enable ingress

helm upgrade -i --wait \
    --namespace cert-manager \
    --create-namespace \
    --set installCRDs=true \
    cert-manager jetstack/cert-manager

helm repo add harbor https://helm.goharbor.io
helm upgrade -i harbor harbor/harbor \
  --set expose.type=ingress \
  --set persistence.enabled=false \
  --set expose.tls.enabled=true

setup /etc/hosts file:
core.harbor.domain
notary.harbor.domain

login at https://core.harbor.domain/ with default username & pass:
        username: admin
        password: Harbor12345

inspect generated certs in browser or with cert tool, check that their SAN is only
for core.harbor.domain, download and import into system cert store (don't forget to remove later). E.g:
sudo cp core-harbor-domain-chain.pem /usr/local/share/ca-certificates/core-harbor-domain-chain.crt
sudo update-ca-certificates

restart docker (minikube start again), so docker loads new certs from OS store

test harbor:
docker tag alpine core.harbor.domain/library/alpine
docker push core.harbor.domain/library/alpine

test kwctl:
kwctl push ~/.cache/kubewarden/store/registry/ghcr.io/kubewarden/policies/pod-privileged:v0.2.1 registry://core.harbor.domain/library/pod-privileged:v0.2.1

kubewarden / kwctl