AKS-Engine fetchs calico-typha-horizontal-autoscaler from the wrong/blocked/global repository and not the Chinese mirror

[AhmedMoAli]

Hello,

After moving to AKS-Engine we noticed that AKS-Engine tries to fetch the docker image for 'calico-typha-horizontal-autoscaler' pod from 'k8s.gcr.io' which is blocked inside china and hence we get ImagePullBackOff error , this is not configurable at the moment and we need to do manual step during our deployment to re-apply the calico-daemonset.yaml and change the image location to point to the Chinese mirror.

  **FROM**: image:_ k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
  **TO**: image: gcr.azk8s.cn/google_containers/cluster-proportional-autoscaler-amd64:1.1.2-r2

[andyzhangx]

Thanks for the feedback. @jackfrancis @CecileRobertMichon You could see all yaml files under https://github.com/Azure/aks-engine/tree/master/parts/k8s/addons are using images like k8s.gcr.io/..., is there a way to make it configurable, in Azure China, it should be gcr.azk8s.cn/google_containers/...

https://github.com/Azure/aks-engine/blob/master/parts/k8s/addons/kubernetesmasteraddons-calico-daemonset.yaml#L652

[AhmedMoAli]

Any feedback?

[andyzhangx]

@jackfrancis @CecileRobertMichon all the images under parts/k8s/addons are hardcorded, it does not work in Azure China, do we have a way to replace these image path in aks-engine? This is a potential issue when rolling new aks-engine on Azure China, it may not work since images from gcr.io, docker.io could not be downloaded on Azure China.

cc @karataliu

~/go/src/github.com/Azure/aks-engine/parts/k8s/addons# grep image: ./* -R
./1.10/kubernetesmasteraddons-calico-daemonset.yaml:      - image: calico/typha:v3.5.0
./1.10/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/cni:v3.5.0
./1.10/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/node:v3.5.0
./1.10/kubernetesmasteraddons-calico-daemonset.yaml:      - image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
./1.11/kubernetesmasteraddons-calico-daemonset.yaml:      - image: calico/typha:v3.5.0
./1.11/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/cni:v3.5.0
./1.11/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/node:v3.5.0
./1.11/kubernetesmasteraddons-calico-daemonset.yaml:      - image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
./1.6/kubernetesmasteraddons-kubernetes-dashboard-deployment.yaml:        image: <img>
./1.6/kubernetesmasteraddons-calico-daemonset.yaml:          image: quay.io/calico/node:v2.4.1
./1.6/kubernetesmasteraddons-calico-daemonset.yaml:          image: quay.io/calico/cni:v1.10.0
./1.7/kubernetesmasteraddons-calico-daemonset.yaml:      - image: calico/typha:v3.5.0
./1.7/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/cni:v3.5.0
./1.7/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/node:v3.5.0
./1.7/kubernetesmasteraddons-calico-daemonset.yaml:      - image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
./1.8/kubernetesmasteraddons-calico-daemonset.yaml:      - image: calico/typha:v3.5.0
./1.8/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/cni:v3.5.0
./1.8/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/node:v3.5.0
./1.8/kubernetesmasteraddons-calico-daemonset.yaml:      - image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
./1.9/kubernetesmasteraddons-calico-daemonset.yaml:      - image: calico/typha:v3.5.0
./1.9/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/cni:v3.5.0
./1.9/kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/node:v3.5.0
./1.9/kubernetesmasteraddons-calico-daemonset.yaml:      - image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
./coredns.yaml:        image: <img>
./kubernetesmasteraddons-azure-npm-daemonset.yaml:          image: containernetworking/azure-npm:v1.0.18
./kubernetesmasteraddons-calico-daemonset.yaml:      - image: calico/typha:v3.5.0
./kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/cni:v3.5.0
./kubernetesmasteraddons-calico-daemonset.yaml:        image: calico/node:v3.5.0
./kubernetesmasteraddons-calico-daemonset.yaml:      - image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
./kubernetesmasteraddons-cilium-daemonset.yaml:  sidecar-istio-proxy-image: "cilium/istio_proxy"
./kubernetesmasteraddons-cilium-daemonset.yaml:        image: docker.io/cilium/cilium:v1.4
./kubernetesmasteraddons-cilium-daemonset.yaml:        image: docker.io/cilium/cilium-init:2018-10-16
./kubernetesmasteraddons-cilium-daemonset.yaml:        image: docker.io/cilium/operator:latest
./kubernetesmasteraddons-cilium-daemonset.yaml:        image: docker.io/cilium/cilium-etcd-operator:v2.0.5
./kubernetesmasteraddons-flannel-daemonset.yaml:        image: quay.io/coreos/flannel:v0.8.0-amd64
./kubernetesmasteraddons-flannel-daemonset.yaml:        image: quay.io/coreos/flannel:v0.10.0-amd64

[CecileRobertMichon]

We can use the same logic as the other addons, see https://github.com/Azure/aks-engine/blob/master/parts/k8s/containeraddons/kubernetesmasteraddons-blobfuse-flexvolume-installer.yaml#L21 for an example. https://github.com/Azure/aks-engine/blob/master/pkg/engine/engine.go#L678 is the function for "ContainerImage".

[andyzhangx]

@CecileRobertMichon thanks, I will the necessary config changes for Azure China later.

[AhmedMoAli]

Thanks for your prompt response, just one thing, when will this be available? We are expected to go live in June

[andyzhangx]

@AhmedMoAli sure, there is a prompt PR: https://github.com/Azure/aks-engine/pull/1089

[andyzhangx]

FYI. This issue is fixed in v0.35.1. Let's still keep this issue open since network cni plugin: cilium-daemonset, flannel-daemonset are not working on Azure China, need same fix