deploy pod from docker registry in same k8s but service name cannot be reach...

I got issue for registry host name reach... When I use this docker-registry in KinD 3 clusters(1 control plane and two worker).

and push image from local via...(I map registry 5000 port to registry.k8s.com via ingress)

docker push registry.k8s.com/grpc-server:V1.2
The push refers to repository [registry.k8s.com/grpc-server]
55344028772e: Pushed
d319ed48691a: Pushed
6470ba8155ed: Pushed
c7bd51621c7a: Pushed
02948dacdd5e: Pushed
c2513ca213d4: Pushed
f618de1e6ce3: Pushed
53cb729241a4: Pushed
c3c01c74818a: Pushed
f83139632251: Pushed
b6f786c730a9: Pushed
63a6bdb95b08: Pushed
8d3ac3489996: Pushed
V1.2: digest: sha256:63586fceda317419374bdabbfd915d8612a6a623fb0bdd4eddd5167fe3d52817 size: 3035

It is successfully and check repository

curl registry.k8s.com/v2/grpc-server/tags/list
{"name":"grpc-server","tags":["V1.2"]}

So image is existed.

and my registry vervice is

kubectl get svc/my-docker-registry
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
my-docker-registry   ClusterIP   10.96.47.9   <none>        5000/TCP   47m

and I want to run this images as pod... kubectl run test-pod --image=my-docker-registry:5000/grpc-server:V1.2 --restart=Never

Then I got fail for deployment

kubectl get pods/test-pod
NAME       READY   STATUS             RESTARTS   AGE
test-pod   0/1     ImagePullBackOff   0          18s

Then I check pod logs...

kubectl describe  pod test-pod
Name:         test-pod
Namespace:    default
Priority:     0
Node:         kind-worker2/172.19.0.3
Start Time:   Tue, 02 Aug 2022 16:17:44 +0800
Labels:       run=test-pod
Annotations:  <none>
Status:       Pending
IP:           10.244.2.12
IPs:
  IP:  10.244.2.12
Containers:
  test-pod:
    Container ID:
    Image:          my-docker-registry:5000/grpc-server:V1.2
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mvf72 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-mvf72:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  53s                default-scheduler  Successfully assigned default/test-pod to kind-worker2
  Normal   BackOff    24s (x2 over 52s)  kubelet            Back-off pulling image "my-docker-registry:5000/grpc-server:V1.2"
  Warning  Failed     24s (x2 over 52s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    12s (x3 over 53s)  kubelet            Pulling image "my-docker-registry:5000/grpc-server:V1.2"
  Warning  Failed     12s (x3 over 52s)  kubelet            Failed to pull image "my-docker-registry:5000/grpc-server:V1.2": rpc error: code = Unknown desc = failed to pull and unpack image "my-docker-registry:5000/grpc-server:V1.2": failed to resolve reference "my-docker-registry:5000/grpc-server:V1.2": failed to do request: Head "https://my-docker-registry:5000/v2/grpc-server/manifests/V1.2": dial tcp: lookup my-docker-registry on 192.168.65.2:53: no such host
  Warning  Failed     12s (x3 over 52s)  kubelet            Error: ErrImagePull

It look like service name cannot be reach in deploy process. I did something wrong? or do I need to config more from this?

Any information will be appreciated.

I found the solution by my self, let me explain... Since deployment pull container by docker which is out of k8s DNS network, so that is why service name cannot be reach. so we cannot use service name.

the solution is ... append following in KinD config file.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
    endpoint = ["http://${host_name}:5000"]

${host_name} is local machine host name, it can be found via "uname -a" to get. After KinD lunch ready, then install docker registry in KinD and also use forward port to 5000 as below:

kubectl port-forward --namespace default svc/docker-registry 5000:5000

if port-forward is not prefer, it need to open port 5000 in KinD and use ingress to expose 5000 mapping to registry service.

then using localhost:5000/${container_name}:${tag_name} for deployment. kubectl run test-pod --image=localhost:5000/${container_name}:${tag_name} --restart=Never then everything is corrected.

Thanks.

twuni / docker-registry.helm

deploy pod from docker registry in same k8s but service name cannot be reach... #66