search

syndesisio / syndesis

This project is archived. A flexible, customizable, open source platform that provides core integration capabilities as a service.
https://syndesis.io/
Apache License 2.0
597 stars 203 forks source link

Upgrade 1.12 to 1.13 doesn't work #9751

Closed mkralik3 closed 3 years ago

mkralik3 commented 3 years ago

This is a...


[ ] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Documentation issue or request

Description

Upgrade Syndesis 1.12.1 to 1.13.0-20211011 doesn't work. The pod db-backup- is not able to be initialized successfully due to this error:

Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Scheduled       27s   default-scheduler  Successfully assigned mkralik/db-backup-1633952344-w6z94 to crc-scmfr-master-0
  Normal   AddedInterface  24s   multus             Add eth0 [10.217.0.85/23] from openshift-sdn
  Normal   Pulling         24s   kubelet            Pulling image "crunchydata/crunchy-pgdump:centos7-12.5-4.5.1"
  Normal   Pulled          5s    kubelet            Successfully pulled image "crunchydata/crunchy-pgdump:centos7-12.5-4.5.1" in 18.667838578s
  Normal   Created         5s    kubelet            Created container backup-db-compiler
  Normal   Started         5s    kubelet            Started container backup-db-compiler
  Normal   Pulling         5s    kubelet            Pulling image "centos:7"
  Warning  Failed          4s    kubelet            Failed to pull image "centos:7": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Invalid Username or Password
  Warning  Failed          4s    kubelet            Error: ErrImagePull
  Normal   BackOff         3s    kubelet            Back-off pulling image "centos:7"
  Warning  Failed          3s    kubelet            Error: ImagePullBackOff

Tried on OCP 4.6, OCP 4.8 and CRC 1.33.1 (OCP 4.8.12).

zregvart commented 3 years ago

I'm assuming that the download rate limit was hit. Perhaps creating a pull secret would help.

mkralik3 commented 3 years ago

The syndesis-pull-secret with docker credentials was there from previous Syndesis 1.12 . I have tried to link the pull secret with the new operator (1.13) manually after the operator was installed

oc scale deployment syndesis-operator --replicas 0
oc secrets link syndesis-operator syndesis-pull-secret --for=pull
oc scale deployment syndesis-operator --replicas 1

but it doesn't resolve the problem.

But, if the rate limit was hit, the previous image (crunchydata/crunchy-pgdump:centos7-12.5-4.5.1) wouldn't be downloaded as well (but it was pulled successfully according to the log). Also, the CRC was running on my local machine where I am able to pull any image from docker hub so in the CRC shouldn't be a problem with the docker download rate limit as well.

zregvart commented 3 years ago

It could be that for such a pull specification OpenShift tries to access a different registry using a different username. It could be that it you're logged in to Docker Hub locally. It could be that at that exact time the request limit was reached.

Either way it looks like something we can workaround at best not an issue we can solve.

We can for instance change this:

https://github.com/syndesisio/syndesis/blob/0d11c6f7d1c7e52ef3a48cb8f4711e0385a35ef8/install/operator/build/conf/config.yaml#L78

To quay.io/centos/centos:7. And at the same time point as many pull specifications as we can away from Docker Hub.

You can test this by changing the config.yaml manually via --operator-config /path/tomy-config.yml.

mkralik3 commented 3 years ago

It looks that the OCP is trying to download centos:7 from some different repository, but, since the log doesn't show the full path of repository, only Pulling image "centos:7", it is hard to say from where. (or, when the repository name is not specified, it generates a different path under the hood, as the podman in my local machine (docker.io/library/centos:7))

When I explicitly specified docker.io/library/centos:7 in the config file, the OCP is able to download it (the image is already on machine).

 Normal  Pulled          4s    kubelet            Container image "docker.io/library/centos:7" already present on machine

I don't know why the OCP4 behaves this ways but to prevent these situations, we probably should add the full path (with repo name) to the image definition.

mkralik3 commented 3 years ago

@zregvart The error message invalid username/password: unauthorized: is misleading since when I try to pull an image from inexistent repository, it is also shown (not that that image/namespace doesn't exist) e.g.

[mkralik@fedora operators]$ docker pull docker.io/incorrectnamespace/incorrectimage:nolatest
Trying to pull docker.io/incorrectnamespace/incorrectimage:nolatest...
Error: initializing source docker://incorrectnamespace/incorrectimage:nolatest: reading manifest nolatest in docker.io/incorrectnamespace/incorrectimage: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
zregvart commented 3 years ago

Awesome, thanks for looking into this more deeply @mkralik3, I think we make a simple fix because of this.