Add custom trusted CA when connecting through HTTP Proxy with Deep Packet Inspection (MITM Certificate)

[flopes89]

When trying to run any kubectl command, I receive

Unable to connect to the server: x509: certificate signed by unknown authority

I suspect this is because our company uses a HTTP Proxy that does Deep Paket Inspection on HTTPS Proxy, which means it injects its own certificate and effectively acts as a MITM.

This trips up kubectl when connecting to our cluster. I want to avoid using insecure flags like --insecure-skip-tls-verify and rather import the CA somwhere so it will always work.

Where do I need to import the CA to in order for kubectl to accept these connections?

[flopes89]

In the meantime, I have already added the CA Certificate in question to the /usr/share/ca-certificates folder and ran sudo dpkg-reconfigure ca-certificates . However, kubectl still refuses to connect.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[eddiezane]

If this issue still persists I believe it might be a better question for Stack Overflow as kubectl will just use system ca's and certs.

/close

[k8s-ci-robot]

@eddiezane: Closing this issue.

In response to [this](https://github.com/kubernetes/kubectl/issues/843#issuecomment-662699301): >If this issue still persists I believe it might be a better question for Stack Overflow as kubectl will just use system ca's and certs. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[projx]

Sorry to drag open an old issue - But you state kubectl will use OS known CA's, is this the case for Windows?

We use SSL decryption on our Firewalls, generally the process is completely transparent, as user's laptops trust our root CA, but when kubectl is used on these same laptop to connect to GKE, the requests fail.

[freyesicp]

@projx were you able to find a solution for the CA?

[toreonify]

If this issue still persists I believe it might be a better question for Stack Overflow as kubectl will just use system ca's and certs.

/close

If it uses system CA, why it still fails?

[localadmin@k8s-master ~]$ ls -la /etc/pki/ca-trust/source/anchors/
total 12
drwxr-xr-x 2 root root 4096 Apr  4 05:28 .
drwxr-xr-x 4 root root 4096 Sep 13  2023 ..
-rw-r--r-- 1 root root 1418 Apr  4 05:27 ans-usergate.crt
[localadmin@k8s-master ~]$ sudo update-ca-trust
[localadmin@k8s-master ~]$ sudo kubeadm init --pod-network-cidr=10.16.0.0/20
I0404 05:49:49.086846    5216 version.go:256] remote version is much newer: v1.29.3; falling back to: stable-1.28
[init] Using Kubernetes version: v1.28.8
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-apiserver:v1.28.8: output: E0404 05:49:50.781661    5300 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = pinging container registry registry.k8s.io: Get \"https://registry.k8s.io/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority" image="registry.k8s.io/kube-apiserver:v1.28.8"
time="2024-04-04T05:49:50Z" level=fatal msg="pulling image: pinging container registry registry.k8s.io: Get \"https://registry.k8s.io/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
, error: exit status 1

curl successfully receives response from registry, btw:

[localadmin@k8s-master ~]$ curl -v https://registry.k8s.io/v2/
* Host registry.k8s.io:443 was resolved.
* IPv6: 2600:1901:0:bbc4::
* IPv4: 34.96.108.209
*   Trying 34.96.108.209:443...
* Connected to registry.k8s.io (34.96.108.209) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /usr/share/ca-certificates/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=registry.k8s.io
*  start date: Mar  3 19:21:51 2024 GMT
*  expire date: Jun  1 20:17:46 2024 GMT
*  subjectAltName: host "registry.k8s.io" matched cert's "registry.k8s.io"
*  issuer: C=RU; L=Novosibirsk; O=UserGate; CN=ugutm_16E525C4D1FCBB18_ca; emailAddress=sales@usergate.com
*  SSL certificate verify ok.
*   Certificate level 0: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET /v2/ HTTP/1.1
> Host: registry.k8s.io
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< X-Tracking-Ref: <0.13423.4931>
< X-Execution-Time: 96176
< Connection: keep-alive
< Keep-Alive: timeout=3, max=5
< docker-distribution-api-version: registry/2.0
< x-cloud-trace-context: 4db9330ab917cca672c72ec4352ed961
< Date: Thu, 04 Apr 2024 05:53:56 GMT
< Content-Type: text/html
< Server: Google Frontend
< Via: 1.1 google, 1.1 google
< Content-Length: 0
< 
* Connection #0 to host registry.k8s.io left intact