Error in pulling jupyterhub image

[bpfrd]

Hello,

I keep getting an error while it's pulling jupyterhub image.

kubectl get po -w
NAME                    READY   STATUS              RESTARTS   AGE
hub-54bb74b68-pgkzd     0/1     ContainerCreating   0          49s
proxy-6965496d4-qpfpj   1/1     Running             0          49s
hub-54bb74b68-pgkzd     0/1     ErrImagePull        0          76s
hub-54bb74b68-pgkzd     0/1     ImagePullBackOff    0          90s

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  3m5s                default-scheduler  Successfully assigned default/hub-54bb74b68-pgkzd to minikube
  Normal   Pulling    95s (x2 over 3m4s)  kubelet            Pulling image "cerit.io/hubs/nbgrader-hub:10-11-2023"
  Warning  Failed     20s (x2 over 109s)  kubelet            Failed to pull image "cerit.io/hubs/nbgrader-hub:10-11-2023": Error response from daemon: Get "https://cerit.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     20s (x2 over 109s)  kubelet            Error: ErrImagePull
  Normal   BackOff    9s (x2 over 109s)   kubelet            Back-off pulling image "cerit.io/hubs/nbgrader-hub:10-11-2023"
  Warning  Failed     9s (x2 over 109s)   kubelet            Error: ImagePullBackOff

When I try to build the images/hub.Dockerfile I get the below error:

[+] Building 0.2s (2/3)                                                                                                                          docker:default
[+] Building 181.1s (5/10)                                                                                                                       docker:default
 => [internal] load .dockerignore                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                            0.0s
 => [internal] load build definition from Dockerfile                                                                                                       0.0s
 => => transferring dockerfile: 642B                                                                                                                       0.0s
 => [internal] load metadata for docker.io/jupyterhub/k8s-hub:3.0.3                                                                                        0.5s
 => CACHED [1/7] FROM docker.io/jupyterhub/k8s-hub:3.0.3@sha256:8b0c02ef8a036a119ef260469b934b314eb49ad9c794911fc4471f2ec888f644                           0.0s
 => ERROR [2/7] RUN apt-get update && apt-get install vim procps fakeroot gcc -y                                                                         180.5s
------
 > [2/7] RUN apt-get update && apt-get install vim procps fakeroot gcc -y:
60.32 Err:1 http://deb.debian.org/debian bullseye InRelease
60.32   Temporary failure resolving 'deb.debian.org'
120.4 Err:2 http://deb.debian.org/debian-security bullseye-security InRelease
120.4   Temporary failure resolving 'deb.debian.org'
180.4 Err:3 http://deb.debian.org/debian bullseye-updates InRelease
180.4   Temporary failure resolving 'deb.debian.org'
180.4 Reading package lists...
180.4 W: Failed to fetch http://deb.debian.org/debian/dists/bullseye/InRelease  Temporary failure resolving 'deb.debian.org'
180.4 W: Failed to fetch http://deb.debian.org/debian-security/dists/bullseye-security/InRelease  Temporary failure resolving 'deb.debian.org'
180.4 W: Failed to fetch http://deb.debian.org/debian/dists/bullseye-updates/InRelease  Temporary failure resolving 'deb.debian.org'
180.4 W: Some index files failed to download. They have been ignored, or old ones used instead.
180.4 Reading package lists...
180.5 Building dependency tree...
180.5 Reading state information...
180.5 E: Unable to locate package procps
180.5 E: Unable to locate package fakeroot
180.5 E: Unable to locate package gcc
------
Dockerfile:3
--------------------
   1 |     FROM jupyterhub/k8s-hub:3.0.3
   2 |     USER root
   3 | >>> RUN apt-get update && apt-get install vim procps fakeroot gcc -y
   4 |     RUN pip install jupyterhub-nativeauthenticator nbgrader https://github.com/CERIT-SC/nbgrader-k8s/releases/download/v0.0.1/nbgrader_k8s_exchange-0.0.1.tar.gz
   5 |
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install vim procps fakeroot gcc -y" did not complete successfully: exit code: 100

best

[KrKOo]

Hey @bpfrd did you have a stable network connection when trying to pull/build the image? I was able to reproduce the same error when I turned off networking on my computer. 😉

[bpfrd]

Hi, Thanks for your repsonse. I think the network is stable on my server. here is the result of pulling an image.

docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 719385e32844: Pull complete Digest: sha256:c79d06dfdfd3d3eb04cafd0dc2bacab0992ebc243e083cabe208bac4dd7759e0 Status: Downloaded newer image for hello-world:latest

Hello from Docker! This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64)
3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/

For more examples and ideas, visit: https://docs.docker.com/get-started/

[KrKOo]

And what if you try this? docker pull cerit.io/hubs/nbgrader-hub:10-11-2023

[bpfrd]

it works.

[KrKOo]

Well... it it works through docker, then it should also work in your K8s cluster. If it doesn't, then the issue is most probably somewhere else, not in the image/image repo. From the docker build logs you posted it looks like network or DNS problems.

[bpfrd]

Yes, the problem is pulling image in minikube. Here i ssh into minikube and try to pull image but it fails:

$minikube ssh docker@minikube:~$ docker@minikube:~$ docker run hello-world Unable to find image 'hello-world:latest' locally docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See 'docker run --help'.

Would you have any idea how to fix it?

[bpfrd]

there was a problem with docker. now it works. But I have another question.

I have set up the ingress as in the example: ingress:

Example ingress configuration

enabled: true
hosts:
  - nbgrader.example.com #replaced by my own domain
tls:
  - hosts:
      - nbgrader.example.com
    secretName: nbgrader-example-com-tls

but proxy-public is still of type clusterip and I can not access the service using in the browser

[KrKOo]

It is the correct behavior for the "proxy-public" service to be ClusterIP. If you are running it on minikube, then I recommend checking out this: https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/

[bpfrd]

Thanks for your suggestion. I should have enabled ingress addons. But now the ingress IP is different from the IP that the host points to. Is it possible to set specific IP to ingress?

[KrKOo]

For testing purposes, isn't it easier to just modify the IP in /etc/hosts?

[bpfrd]

I'm not very familiar with networking. The problem is I would like to deploy it using minikube and jupyterhub should be accessible from browser. The domain resolves to some IP starting from 10.xxx on the server while ingress address for the same host is another IP starting from 192.xxx. I think if ingress address could be the same address 10.xxx it would work. How is it possible?

[KrKOo]

It should work with the 192.xxx. Just put the following line in the /etc/hosts file:

192.x.x.x     your.domain.com

Where your.domain.com is the same, as you've set in the ingress configuration. Then you should be able to access jupyterhub on this domain.

[bpfrd]

I added 192.xxx mydomain but it didn't work. do i need to reboot the server? maybe this information helps.

ping mydomain (10.xxx) from local computer works $kubectl get ingress gives HOST=mydomain and address 192.xxx

the server is remote and different from my local computer. I tried to put c.JupyterHub.bind_url=http://10.xxx:80 but it didn't work either.

[bpfrd]

more info:

$kubectl get ingress
NAME         CLASS    HOSTS                 ADDRESS        PORTS     AGE
jupyterhub   <none>   domain.com  my_minikube_ip(192.xxx)   80, 443   3h56m

$kubectl get ingress -o yaml
...
spec:
    rules:
    - host: domain.com
      http:
        paths:
        - backend:
            service:
              name: proxy-public
              port:
                name: http
          path: /
          pathType: Prefix
    tls:
    - hosts:
      - domain.com
      secretName: my-tls-secret
  status:
    loadBalancer:
      ingress:
      - ip: my_minikube_ip

and the domain.com resolves to 10.xxx

[KrKOo]

What server? Minikube is intended to be used only in a local environment. Are you trying to run minikube on a remote server? Please do not try to run a production application on minikube.

It can probably be done by port-forwarding/tunneling/something, but it is definitely not what minikube is designed for.

[bpfrd]

I see. Even if we allocate enough resources CPU and memory and the class is small, I shouldn't run minikube in production? What would you suggest? (I mean own server not a cloud-based).

[KrKOo]

You can check k3s, MicroK8s or kubeadm for example. But they are a little harder to setup than minikube.

I'm closing the issue, since the discussion diverged from the original topic.