search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.6k stars 4.89k forks source link

crio: Unable to pull images form internal registry #10171

Open kameshsampath opened 3 years ago

kameshsampath commented 3 years ago 
minikube version: v1.16.0
commit: 9f1e482427589ff8451c4723b6ba53bb9742fbb1

Steps to reproduce the issue:

  1. Start Minikube as 
    minikube start -p $PROFILE_NAME \
--memory=$MEMORY --cpus=$CPUS \
--disk-size=50g \
--insecure-registry='10.0.0.0/24'
  2. Apply the registry and registry-aliases add-on
  3. Build and push an image to the Minikube registry as example.com/demo/greeter
  4. Deploy a pod using the command 
    kubectl run demo-greeter -n tektontutorial \
--generator='run-pod/v1' \
--image='example.com/demo/greeter' && \
kubectl expose pod demo-greeter -n tektontutorial --port 8080 --type=NodePort

The kubectl get events shows the crashed pod with the following logs:

Full output of failed command: 

0s          Warning   Failed      pod/demo-greeter                        Failed to pull image "registry.minikube/rhdevelopers/tekton-tutorial-greeter": rpc error: code = Unknown desc = error pinging docker registry registry.minikube: Get https://registry.minikube/v2/: dial tcp 10.101.166.84:443: connect: connection refused

With --insecure-registry specified, the pod still tries to resolve the url as https than http

CC: @afbjorklund

kameshsampath commented 3 years ago

some more contextual info

minikube ssh -- cat /etc/hosts

127.0.0.1       localhost
127.0.1.1 tektontutorial
192.168.64.1    host.minikube.internal
192.168.64.20   control-plane.minikube.internal
10.101.166.84   example.org
10.101.166.84   example.com
10.101.166.84   test.com
10.101.166.84   test.org
10.101.166.84   registry.minikube

kubectl get svc -n kube-system

AME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   79m
registry   ClusterIP   *10.101.166.84*   <none>        80/TCP,443/TCP           78m

kubectl get cm coredns -n kube-system -o yaml | yq eval '.data' -

Corefile: |-
  .:53 {
      errors
       rewrite name example.org  registry.kube-system.svc.cluster.local
  rewrite name example.com  registry.kube-system.svc.cluster.local
  rewrite name test.com  registry.kube-system.svc.cluster.local
  rewrite name test.org  registry.kube-system.svc.cluster.local
  rewrite name registry.minikube  registry.kube-system.svc.cluster.local

      health {
         lameduck 5s
      }
      ready
      kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods insecure
         fallthrough in-addr.arpa ip6.arpa
         ttl 30
      }
      prometheus :9153
      forward . /etc/resolv.conf {
         max_concurrent 1000
      }
      cache 30
      loop
      reload
      loadbalance
  }
afbjorklund commented 3 years ago

Worked here.

  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  4s               default-scheduler  Successfully assigned default/busybox to minikube
  Normal   Pulled     4s               kubelet            Successfully pulled image "registry.minikube/busybox" in 16.245797ms
  Normal   Pulling    3s (x2 over 4s)  kubelet            Pulling image "registry.minikube/busybox"
  Normal   Created    3s (x2 over 4s)  kubelet            Created container busybox
  Normal   Started    3s (x2 over 3s)  kubelet            Started container busybox
  Normal   Pulled     3s               kubelet            Successfully pulled image "registry.minikube/busybox" in 18.747139ms
  Warning  BackOff    1s (x2 over 2s)  kubelet            Back-off restarting failed container

You might want to double-check minikube ssh docker info:

 Insecure Registries:
  10.96.0.0/12
  10.0.0.0/24
  127.0.0.0/8
kameshsampath commented 3 years ago

that sound weird ! was it on macOS ?

Let me try again on a fresh instance, do you see any issue with minikube start command ??

kameshsampath commented 3 years ago

after a bit of debug @afbjorklund , this happens only with container-runtime=cri-o, if I use container-runtime=docker then all works well. Wondering something to do with internal work on cri-o?

medyagh commented 3 years ago

@kameshsampath interesting do we still have this issue on latest minikube ? I wonder if this issue happens if we specify a different CNI ?

kameshsampath commented 3 years ago

still get the same issue with crio, not sure what CNI driver I can use on macOS.

From: Medya Ghazizadeh notifications@github.com Sent: Thursday, March 4, 2021 1:03 AM To: kubernetes/minikube minikube@noreply.github.com Cc: Kamesh Sampath kamesh.sampath@hotmail.com; Mention mention@noreply.github.com Subject: Re: [kubernetes/minikube] crio: Unable to pull images form internal registry (#10171)

@kameshsampathhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkameshsampath&data=04%7C01%7C%7C00d057535cea4f79d76208d8de7b3c39%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637503968168281687%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VTpOI4Z98MweIKCzJepHxP1nzePl%2BSF0aZTki7FhKmE%3D&reserved=0 interesting do we still have this issue on latest minikube ? I wonder if this issue happens if we specify a different CNI ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fminikube%2Fissues%2F10171%23issuecomment-789999613&data=04%7C01%7C%7C00d057535cea4f79d76208d8de7b3c39%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637503968168291684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nPweTXdxmB6yEC15VbgvkkoExj9gfuV0G7mGH%2B4Zgu0%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHHMILTS6VCW3ABYI2ORALTB2FI3ANCNFSM4WIK3JNA&data=04%7C01%7C%7C00d057535cea4f79d76208d8de7b3c39%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637503968168301691%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=r13cQSiqkXiJOP2%2BFJNBCqWctcAQST4M55tCVf%2BtNsE%3D&reserved=0.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

k8s-triage-robot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 3 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/minikube/issues/10171#issuecomment-913034015): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
ricardozanini commented 8 months ago

Just for your information, I got in this issue today with the latest Minikube version. Same setup MacOS, podman, and cri-o. I also started the cluster with the flag --insecure-registry=10.0.0.0/24. The kubelet controller insists on using HTTPS instead of HTTP.

The problem is that the registry service is created exposing the ports 80:5000 and 443:443. But the registry pod, only exposes 5000. So obviously, kubelet or any client won't be able to pull from 443.

I think this is a bug @kameshsampath @medyagh

Cheers!

ricardozanini commented 8 months ago

/reopen

k8s-ci-robot commented 8 months ago

@ricardozanini: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes/minikube/issues/10171#issuecomment-2032973386): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
medyagh commented 8 months ago

he latest Minikube version. Same setup MacOS, podman, and cri-o. I also started the cluster with the flag --insecure-registry=10.0.0.0/24. The kubelet controller insists on using HTTPS instead of HTTP.

The problem is that the registry service is created exposing the ports 80:5000 and 443:443. But the registry pod, only exposes 5000. So obviously, kubelet or any client won't be able to pull from 443.

ricardozanini would you be interested to make a PR to fix this ?

of the issue is Container needs to open a non 443 port you can do that in the Kic Package

ricardozanini commented 8 months ago

@medyagh you bet I would! :)

I'll take a look later today and send a PR.