[MinIO addon] console pod goes into ImagePullBackOff

deusebio commented 5 months ago

Summary

When enabling MinIO addon on MicroK8s 1.28-strict, the console pod never starts because of a ImagePullBackOff error due to the fact that it is unable to download the minio/console image from docker. I checked on DockerHub, and indeed, the minio/console page (which is indexed by google) does not seem to exist anymore.

Here is the kubectl describe command output

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  38s                default-scheduler  Successfully assigned minio-operator/console-78d567bfc8-vkbhp to pelican
  Normal   Pulling    19s (x2 over 37s)  kubelet            Pulling image "minio/console:v0.20.3"
  Warning  Failed     16s (x2 over 34s)  kubelet            Failed to pull image "minio/console:v0.20.3": failed to pull and unpack image "docker.io/minio/console:v0.20.3": failed to resolve reference "docker.io/minio/console:v0.20.3": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     16s (x2 over 34s)  kubelet            Error: ErrImagePull
  Normal   BackOff    5s (x2 over 34s)   kubelet            Back-off pulling image "minio/console:v0.20.3"
  Warning  Failed     5s (x2 over 34s)   kubelet            Error: ImagePullBackOff

The console pod is then on a ImagePullBackOff error loop

What Should Happen Instead?

The console pod should just start fine.

Reproduction Steps

sudo snap install microk8s --channel 1.28-strict/stable
microk8s enable minio

Introspection Report

inspection-report-20240423_091630.tar.gz

Can you suggest a fix?

I'm wondering whether the MinIO console image has been pulled out from DockerHub or if it got moved. If it was pull out and there is not alternative, we should probably either publish a Canonical supported one to have more control over it or disable the console to be run by default.

Are you interested in contributing with a fix?

If provided with guidance, I could give it a try. But I've never dug in the MicroK8s codebase, so if it's not a good-first issue (and given that this seems a bit critical/impactful to everyone using MinIO), maybe I'm not the best bet.

eaudetcobello commented 5 months ago

Hi @deusebio,

Thanks for reporting this issue. I'll be working on getting this fixed.

nirmesh commented 5 months ago

I am also facing same issue. is there any work around? When will it get fixed?

sennid commented 5 months ago

https://github.com/minio/console/issues/3318 - they no longer upload images for console since it's now part of minio/minio same binary

abctaylor commented 4 months ago

Confirmed I see this too

eaudetcobello commented 4 months ago

Hi everyone,

The minio team has deleted the console image from Dockerhub, which is problematic for the minio addon because it uses v4.5 of the minio-operator which relies on the console image. A temporary workaround is to find another source for the console image (eg. quay.io/minio/console) and modify the deployments so they use this image instead of just minio/console.

For a new cluster, you can replace line 89 of /var/snap/microk8s/common/addons/core/addons/minio/enable with this line: "${SNAP_COMMON}/plugins/kubectl-minio" init --console-image quay.io/minio/console:v0.23.0

Please note that these workarounds are not officially supported by Canonical, and the Microk8s team will be upgrading the minio addon version in the next patch release of Microk8s.

oryza-anggara commented 2 months ago

Is there any update?

How about set the image directly on the ReplicaSets? like.. kubectl set image replicaset.apps/console-78d567bfc8 -n=minio-operator console=quay.io/minio/console:v0.23.0

would it lead to further issue?

cedricve commented 1 month ago

Is there any update?

How about set the image directly on the ReplicaSets? like.. kubectl set image replicaset.apps/console-78d567bfc8 -n=minio-operator console=quay.io/minio/console:v0.23.0

would it lead to further issue?

This didnt work for me.

cedricve commented 1 month ago

Hi @deusebio,

Thanks for reporting this issue. I'll be working on getting this fixed.

Any idea on when this will be resolved? I believe this one is quiet important to support the addon community.

cedricve commented 1 month ago

The thing is that I'm currently unable to install the Minio operator as running it through the preferred installation method of Minio (with Kustomize) also gives an pthread error with microk8s (https://github.com/canonical/microk8s/issues/4550).

  root@microk8s:~# kubectl apply -k github.com/minio/operator\?ref=v6.0.1
  error: failed to run '/snap/microk8s/7040/usr/bin/git fetch --depth=1 https://github.com/minio/operator v6.0.1': /usr/lib/git-core/git-remote-https: symbol lookup error: /snap/core20/current/lib/x86_64-linux-gnu/libpthread.so.0: undefined symbol: __libc_pthread_init, version GLIBC_PRIVATE

I'm now forced to clone the repository manually.

git clone --depth 1 --branch v6.0.1 https://github.com/minio/operator.git && kubectl apply -k operator/

canonical / microk8s