storage-provisioner Pod镜像名称错误

[johntostring]

最近需要一个 v1.19.8 的 K8s 集群进行测试，minikube 成功创建完成后，查看 Pod 发现 storage-provisioner 拉取镜像失败。

重现问题所需的命令：

minikube start \
--kubernetes-version=v1.19.8 \
--image-mirror-country=cn \
--cpus=2 \
--memory=4g \
--driver=parallels

失败的命令的完整输出： 没有失败

minikube logs命令的输出：

😄  Darwin 10.15.6 上的 minikube v1.20.0
❗  Kubernetes 1.19.8 has a known performance issue on cluster startup. It might take 2 to 3 minutes for a cluster to start.
❗  For more information, see: https://github.com/kubernetes/kubeadm/issues/2395
✨  根据用户配置使用 parallels 驱动程序
✅  正在使用镜像存储库 registry.cn-hangzhou.aliyuncs.com/google_containers
👍  Starting control plane node minikube in cluster minikube
🔥  Creating parallels VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
🐳  正在 Docker 20.10.6 中准备 Kubernetes v1.19.8…
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-minikube/storage-provisioner:v5 (global image repository)
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

使用的操作系统版本： macOS Catalina 10.15.6 (19G2021)

查看 Pod

>kubectl get pod -A
NAMESPACE     NAME                               READY   STATUS             RESTARTS   AGE
kube-system   coredns-6c76c8bb89-qsvmm           1/1     Running            0          99s
kube-system   etcd-minikube                      1/1     Running            0          117s
kube-system   kube-apiserver-minikube            1/1     Running            0          117s
kube-system   kube-controller-manager-minikube   1/1     Running            0          117s
kube-system   kube-proxy-7fmdh                   1/1     Running            0          99s
kube-system   kube-scheduler-minikube            1/1     Running            0          117s
kube-system   storage-provisioner                0/1     ImagePullBackOff   0          115s

>kubectl -n kube-system describe pod storage-provisioner
...
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  2m12s               default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Warning  FailedScheduling  2m12s               default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled         115s                default-scheduler  Successfully assigned kube-system/storage-provisioner to minikube
  Normal   BackOff           33s (x6 over 114s)  kubelet            Back-off pulling image "registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-minikube/storage-provisioner:v5"
  Normal   Pulling           19s (x4 over 115s)  kubelet            Pulling image "registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-minikube/storage-provisioner:v5"
  Warning  Failed            18s (x4 over 115s)  kubelet            Failed to pull image "registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-minikube/storage-provisioner:v5": rpc error: code = Unknown desc = Error response from daemon: pull access denied for registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-minikube/storage-provisioner, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
  Warning  Failed            18s (x4 over 115s)  kubelet            Error: ErrImagePull
  Warning  Failed            5s (x7 over 114s)   kubelet            Error: ImagePullBackOff

SSH 进入 minikube 虚机查看镜像，发现镜像名称 实际是 registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner 并非是 registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-minikube/storage-provisioner

>minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ docker images
REPOSITORY                                                                    TAG        IMAGE ID       CREATED         SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner       v5         6e38f40d628d   3 months ago    31.5MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.19.8    ea03182b84a2   4 months ago    118MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.19.8    9ba91a90b7d1   4 months ago    119MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.19.8    213ae7795128   4 months ago    111MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.19.8    919a3f36437d   4 months ago    46.5MB
registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard                 v2.1.0     9a07b5b4bfac   6 months ago    226MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.4.13-0   0369cf4303ff   10 months ago   253MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.7.0      bfe3a36ebd25   12 months ago   45.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper           v1.0.4     86262685d9ab   15 months ago   36.9MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.2        80d28bedfe5d   16 months ago   683kB

[johntostring]

PR #11785 应该能修复这个问题

[sharifelgamal]

@medyagh did you merge a PR to fix this?

[gaoxinge]

@sharifelgamal Before pr merge, you can try the commands below

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5 registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-minikube/storage-provisioner:v5
minikube image load registry.cn-hangzhou.aliyuncs.com/google_containers/k8s-minikube/storage-provisioner:v5

[sharifelgamal]

This is fixed in 1.23.2