coreos / coreos-kubernetes

CoreOS Container Linux+Kubernetes documentation & Vagrant installers
https://coreos.com/kubernetes/docs/latest/
Apache License 2.0
1.1k stars 466 forks source link

kubelet-wrapper not working with http_proxy environment variables #664

Open jemattson opened 8 years ago

jemattson commented 8 years ago

Kubelet-wrapper is not utilizing the http_proxy or https_proxy environment variables. Our environment requires proxying of all outbound internet connections. As we are utilizing AWS as our cloud provider, all outbound connections return a timeout error in the kubelet service. Connections from this same host, outside of kubelet wrapper are able to successfully use the proxy for internet connectivity.

kubelet.service configuration: [Service] ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests Environment=KUBELET_VERSION=v1.3.4_coreos.0 Environment="http_proxy=http://192.168.213.4:8080" Environment="https_proxy=http://192.168.213.4:8080" Environment="RKT_OPTS=--volume dns,kind=host,source=/etc/resolv.conf --mount volume=dns,target=/etc/resolv.conf" ExecStart=/usr/lib/coreos/kubelet-wrapper --cloud-provider=aws --pod-infra-container-image=703229635267.dkr.ecr.us-east-1.amazonaws.com/google_containers/pause-amd64:3.0 --api-servers=http://127.0.0.1:8080 --network-plugin-dir=/etc/kubernetes/cni/net.d --network-plugin= --register-schedulable=false --allow-privileged=true --config=/etc/kubernetes/manifests --hostname-override=192.168.213.114 --cluster-dns=10.3.0.10 --cluster-domain=cluster.local Restart=always RestartSec=10 [Install] WantedBy=multi-user.target

ERROR in Kubelet service: Sep 13 15:25:25 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci Sep 13 15:25:26 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: image: using image from local store for image name quay.io/coreos/hyperkube:v1.3.4 Sep 13 15:25:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: I0913 15:25:32.398355 8754 docker.go:327] Start docker client with request time Sep 13 15:25:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: W0913 15:25:32.488018 8754 server.go:487] Could not load kubeconfig file /var/l Sep 13 15:25:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: W0913 15:25:32.488403 8754 server.go:448] Could not load kubernetes auth path / Sep 13 15:25:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: I0913 15:25:32.489064 8754 aws.go:604] Zone not specified in configuration file Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: E0913 15:27:32.840088 8754 server.go:293] Failed running kubelet: could not ini Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: caused by: Post https://ec2.us-east-1.amazonaws.com/: dial tcp 54.239.29.8:443: i/ Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: Error: could not init cloud provider "aws": error finding instance i-fbcaa762: err Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: caused by: Post https://ec2.us-east-1.amazonaws.com/: dial tcp 54.239.29.8:443: i/ Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: could not init cloud provider "aws": error finding instance i-fbcaa762: error list Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local kubelet-wrapper[8754]: caused by: Post https://ec2.us-east-1.amazonaws.com/: dial tcp 54.239.29.8:443: i/ Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local systemd[1]: kubelet.service: Unit entered failed state. Sep 13 15:27:32 ip-192-168-213-114.pvdmz.local systemd[1]: kubelet.service: Failed with result 'exit-code'. Sep 13 15:27:42 ip-192-168-213-114.pvdmz.local systemd[1]: kubelet.service: Service hold-off time over, scheduling restart. Sep 13 15:27:42 ip-192-168-213-114.pvdmz.local systemd[1]: Stopped kubelet.service.

alekssaul commented 8 years ago

Can you try setting the http_proxy and https_proxy on the OS level, such as

  - path: "/etc/environment"
    permissions: "0644"
    owner: "root"
    content: |
      http_proxy=http://192.168.213.4:8080
      https_proxy=http://192.168.213.4:8080

also since you are running http proxy behind https proxy you may need to do;

rkt fetch quay.io/coreos/hyperkube/v1.3.4_coreos.0 first

alekssaul commented 7 years ago

@jemattson did you get a chance to try above ? I've done limited testing with it and curious to find out results.