Gitlab registry doesn't work on MicroK8s only

[hiagodotme]

Summary

I have a private registry of images provided by Gitlab. My images have the tag registry.mydomain.com/my-group/my-image:1.0.0.

Registration requires authentication, both login and password. The registry has a valid SSL certificate provided by Let's Encrypt.

On any computer, inside or outside my network, I can do a docker pull registry.mydomain.com/my-group/my-image:1.0.0 after doing a docker login registry.mydomain.com.

Already when I get to MicroK8s, I can't get it to work. I have already done all the procedures described at https://microk8s.io/docs/registry-private. I've also tried creating secrets with kubectl create secret docker-registry... and listing them in my POD specification. But I always get the error:

• Error: ImagePullBackOff

When running a describe on my pod I get the following output:

Events:
   Type Reason Age From Message
   ---- ------ ---- ---- -------
   Normal BackOff 6m57s (x20 over 12m) kubelet Back-off pulling image "registry.mydomain.com/my-group/my-image:1.0.0"
   Normal SandboxChanged 2m46s kubelet Pod sandbox changed, it will be killed and re-created.
   Normal BackOff 85s (x2 over 2m44s) kubelet Back-off pulling image "registry.mydomain.com/my-group/my-image:1.0.0"
   Warning Failed 85s (x2 over 2m44s) kubelet Error: ImagePullBackOff
   Normal Pulling 71s (x3 over 2m45s) kubelet Pulling image "registry.mydomain.com/my-group/my-image:1.0.0"
   Warning Failed 71s (x3 over 2m44s) kubelet Failed to pull image "registry.mydomain.com/my-group/my-image:1.0.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry. mydomain.com/my-group/my-image:1.0.0": failed to resolve reference "registry.mydomain.com/my-group/my-image:1.0.0": failed to authorize: failed to fetch anonymous token : Get "https://gitlab.mydomain.com/jwt/auth?scope=repository%3Amy-group%2Fmy-image%3Apull&service=container_registry": x509: certificate is not valid for any names, but wanted to match gitlab. mydomain.com
   Warning Failed 71s (x3 over 2m44s) kubelet Error: ErrImagePull
   Normal SandboxChanged 32s kubelet Pod sandbox changed, it will be killed and re-created.
   Warning Failed 22s kubelet Failed to pull image "registry.mydomain.com/my-group/my-image:1.0.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mydomain.com/my -group/my-image:1.0.0": failed to resolve reference "registry.mydomain.com/my-group/my-image:1.0.0": failed to authorize: failed to fetch oauth token: Post "https: //gitlab.mydomain.com/jwt/auth": x509: certificate is not valid for any names, but wanted to match gitlab.logical.delivery
   Warning Failed 22s kubelet Error: ErrImagePull
   Normal BackOff 22s kubelet Back-off pulling image "registry.mydomain.com/my-group/my-image:1.0.0"
   Warning Failed 22s kubelet Error: ImagePullBackOff
   Normal Pulling 8s (x2 over 32s) kubelet Pulling image "registry.mydomain.com/my-group/my-image:1.0.0"

One point of attention: in the events is that for some reason it changes the registry domain to "gitlab.mydomain.com/jwt/auth". And it gives a certificate error stating that the certificate found responds to "gitlab.mydomain.com". This is very strange, the only certificate I have is a wildcard.

What Should Happen Instead?

You should download the image and create the container

Reproduction Steps

1. In a gitlab instance, configure image registration.
2. After that, create an image and push it.
3. Test the pull on your machine or any other machine you have access to. (it works)
4. Try using this image on a MicroK8s instance, using the existing documentation for private registration. (does not work)

Can you suggest a fix?

I would like to know if anyone has ever managed to get this to work. And if possible, understand why you were able to pull the image outside the MicroK8s and not inside it.

[hiagodotme]

I also made this report for the Gitlab community:

https://gitlab.com/gitlab-org/gitlab/-/issues/440263

[ktsakalozos]

Hi @hiagodotme, sorry for the late reply. Have you looked into https://microk8s.io/docs/registry-private ?

[hiagodotme]

Hi @hiagodotme, sorry for the late reply. Have you looked into https://microk8s.io/docs/registry-private ?

Hello everything is fine? Yes, I already analyzed it, unfortunately I ended up deploying a docker registry separate from gitlab. Doing it this way, it worked. But now it's an additional service for us to manage, the strange thing is that for some reason the gitlab registry works with the docker cli.

[ruimaciel]

I'm experiencing this issue as well: microk8s is unable to pull container images out of my gitlab container registries.

@ktsakalozos were you able to reproduce this issue from your end?

[sammyeby]

I'm also experiencing the same issue when pulling private gitlab repos in Microk8s. Error: ImagePullBackOff Any updates? Thanks in advance :D

[hiagodotme]

Hi @ruimaciel and @sammyeby, unfortunately I couldn't get it to work with gitlab's private registry. I had some ideas, but as I couldn't dedicate much time to it, I'll tell you what I did:

I ended up uploading a private registry container, and configuring it worked. The bad part is that now I have yet another service to manage and back up.

What I also thought of was trying to create a proxy from Docker's private registry to Gitlab's:

K8s does the pull => (my docker private registry proxy) => (my gitlab proxy).

Unfortunately I couldn't dedicate myself to trying to do it this way, so I created a private registry and pushed directly to it.

[neoaggelos]

Hi all, this looks like an issue with the gitlab registry perhaps not being properly configured? Have you set https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations to point to your gitlab's oauth server?

Also, is this instance using a self-signed certificate? Shot in the dark, but maybe you need something like https://microk8s.io/docs/registry-private#using-a-custom-ca-8?

Overall, this looks like a configuration issue on the GitLab side rather than a microk8s issue.

[ruimaciel]

I don't think it's a configuration issue on GitLab's side. I'm using the repositories hosted by GitLab itself and I'm able to pull container images both with Docker and microk8's own containerd CLI.

Did you managed to reproduce the issue on your side?

Thanks, Rui Maciel

On Tue, Apr 9, 2024, 12:54 Angelos Kolaitis @.***> wrote:

Hi all, this looks like an issue with the gitlab registry perhaps not being properly configured? Have you set https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations to point to your gitlab's oauth server?

This looks like a configuration issue on the GitLab side rather than a microk8s issue.

— Reply to this email directly, view it on GitHub https://github.com/canonical/microk8s/issues/4392#issuecomment-2044707344, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJJILW2QJIMXYY5QVXRJ3Y4PCF7AVCNFSM6AAAAABCWNCWVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBUG4YDOMZUGQ . You are receiving this because you were mentioned.Message ID: @.***>

[sammyeby]

Hi @hiagodotme and @ruimaciel, I finally got it to work on my macBook.

I'm using helm to manage my microk8s cluster.

When I create secrets.yaml manually myself and base64 encode the docker/config.json myself with my registry credential in it ( {"auths":{"registry.gitlab.com":{"auth":"ahahahahahahaha==","username":"xxxx","password":"xxxxxx","email":"myemail@email.com"}}} ) like below,

docker-secrets.yaml

apiVersion: v1 kind: Secret metadata: name: dockercred type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: AJJsjjssha-BASE64-STRING-sjadjjfss==

and install it (with imagePullSecrets in my deployment.yaml file ) via helm, for some reason it doesn't work. I get that ImagePullBackOff unauthorised error. Maybe my encoding was wrong, I don't know.

What worked for me is creating the secret manually in command line (terminal in Mac) like this: microk8s kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

as stipulated in the official document: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-by-providing-credentials-on-the-command-line

Now I get success. It's a bit strange but maybe it can help you as well.

`
Events: Type Reason Age From Message

---

Normal Scheduled 56s default-scheduler Successfully assigned default/myservice-847f77d9d8-mtkl6 to microk8s-vm Normal Pulled 48s kubelet Successfully pulled image "registry.gitlab.com/example/my-service/main" in 5.584s (5.585s including waiting) Normal Pulling 9s (x2 over 54s) kubelet Pulling image "registry.gitlab.com/example/my-service/main" Normal Pulled 7s kubelet Successfully pulled image "registry.gitlab.com/example/my-service/main" in 1.028s (1.029s including waiting) Normal Created 7s (x2 over 48s) kubelet Created container myservice Normal Started 7s (x2 over 47s) kubelet Started container myservice `

[bb4L]

I think i have a similar issue

i did the config in the /var/snap/microk8s/current/args/containerd-template.toml

as described here: https://github.com/containerd/containerd/blob/78d3e205a51ec101f775a43bee6f4fdd8fc6b22b/docs/cri/registry.md

[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.mydomain".auth]
  username = "my-user"
  password = "my-password"

judging from the logs microk8s seems to be able to login but not pull the image

it fails on a url following the pattern https://registry.mydomain/v2/gitlab_project_path/manifests/sha256:SHA

where SHA is the manifest digest of the image shown in gitlab

this request fails with a 404

any ideas what i might be doing wrong? (i double checked the credentials locally, they work)

[hiagodotme]

Hi everyone, I stopped insisting on GitLab containers.

I ended up creating a private registry using a Docker container and had no problems. Unfortunately, I didn't have enough time to keep insisting on it.

[hiagodotme]

Não acho que seja um problema de configuração do lado do GitLab. Estou usando o repositórios hospedados pelo próprio GitLab e eu sou capaz de extrair imagens de contêiner tanto com o Docker quanto com a própria CLI do containerd do microk8. Você conseguiu reproduzir o problema do seu lado? Obrigado Rui Maciel … On Tue, Apr 9, 2024, 12:54 Angelos Kolaitis @.> wrote: Hi all, this looks like an issue with the gitlab registry perhaps not being properly configured? Have you set https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations to point to your gitlab's oauth server? This looks like a configuration issue on the GitLab side rather than a microk8s issue. — Reply to this email directly, view it on GitHub <#4392 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJJILW2QJIMXYY5QVXRJ3Y4PCF7AVCNFSM6AAAAABCWNCWVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBUG4YDOMZUGQ . You are receiving this because you were mentioned.Message ID: @.>

So, in docker it works normally. I authenticate, push and pull normally. Only in microk8s I couldn't do it.

[sammyeby]

Hi @bb4L Since my last comment in April above, I have been using and still use my private GitLab projects containers with Microk8s successfully.

From your comment, you said

_"the logs microk8s seems to be able to login but not pull the image it fails on a url following the pattern https://registry.mydomain/v2/gitlab_project_path/manifests/sha256:SHA"_ with 404 not found.

Doesn't that mean that the image URL is incorrect? Have you checked the GitLab container image naming conventions? Maybe that would help with you because I never use /manifests/sha256:SHA as part of my URLs to pull my images from GitLab registry in my microk8s cluster.

I just follow one of the naming conventions below: https://docs.gitlab.com/ee/user/packages/container_registry/#naming-convention-for-your-container-images

[bb4L]

hi @sammyeby

i forgot to update here that i managed to set it up successfully.

The issue was iirc related to which registry version i used, now i'm using 2.8.3