Rancher install fails on k3s behind corporate proxy

[janeczku]

What kind of request is this (question/bug/enhancement/feature request): Bug

Steps to reproduce (least amount of steps as possible):

1. Deploy an HTTP proxy in the network (e.g. tinyproxy)
2. Install single- or multinode k3s cluster following the HA install docs
3. Configure k3s with the proxy settings in order to allow the pulling of images from Docker Hub:

   
   $ cat /etc/systemd/system/k3s.service.env
   no_proxy=localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
   https_proxy=http://10.135.142.242:8888/
   http_proxy=http://10.135.142.242:8888/

$ systemctl restart k3s

4. Install cert-manager and Rancher [following the steps for Rancher generated certificates or LE](https://rancher.com/docs/rancher/v2.x/en/installation/k8s-install/helm-rancher/)

**Result:**
Rancher Helm install fails with error:

$ helm install rancher rancher-latest/rancher \

--namespace cattle-system \ --set hostname=rancher.my.org Error: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: Unable to connect

Other details that may be helpful: HTTP Proxy log shows that Kubernetes API server uses the Proxy to connect to the cert-manager webhook even though .svc domain is in the NO_PROXY exclude list.

CONNECT   Apr 24 10:40:16 [30184]: Request (file descriptor 7): CONNECT cert-manager-webhook.cert-manager.svc:443 HTTP/1.1
INFO      Apr 24 10:40:16 [30184]: No upstream proxy for cert-manager-webhook.cert-manager.svc
ERROR     Apr 24 10:40:16 [30184]: opensock: Could not retrieve info for cert-manager-webhook.cert-manager.svc
INFO      Apr 24 10:40:16 [30184]: no entity

k3s api-server warning:

Apr 24 11:05:17 jb-k3s-a-01 k3s[29570]: W0424 11:05:17.456815   29570 dispatcher.go:180] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: Unable to connect

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.4.2
• Installation option (single install/HA): HA