This container version is no longer available from the Crunchy Data Developer Program.

[idanl21]

Hey :) im using your Operator for Prod and i started to get that error on my cluster: Error response from daemon: unknown: This container version is no longer available from the Crunchy Data Developer Program. For information on accessing these containers, please contact info@crunchydata.com. Did you deleted those images? using images : registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:centos8-4.7.5 registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.3-4.7.0

[CaptainKro]

I have the same problem. I try to update the version of image with the last example and it's the same. Perhaps it's a registry problem.

All my production cluster is currently broken :-(

[radepajic]

The same applies to me. I have problem with pulling image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-13.8-1

[craigkerstiens]

Sorry to hear about this issue. There has been some discussion of this in past GitHub issues and communications through various channels, but the short answer is that Crunchy Data makes available the last two Postgres versions as part of the Postgres Developer program. We've sent out notifications on our developer mailing list regarding this and would recommend subscribing there.

Developer portal images are not intended for production use. But we very much do not want folks to be in a bad place. If you have feedback on how we could better improve communication would welcome that feedback. At the moment we're exploring how to better enhance and improve the community around this so hopefully that helps in the future.

If you can contact us at info@crunchydata.com we are happy to help you upgrade or if you need support for running in production would be happy to have a conversation around Crunchy Postgres for Kubernetes.

[adriannieto-attechnest]

How this ended up happening? https://www.postgresql.org/support/versioning/ PG13 is still in production cycle...

[keithlayne]

Developer portal images are not intended for production use.

@craigkerstiens - okay. Why do I have them in production? Is it because I don't have a paid subscription? I mean, 4.7.10 was released yesterday, I just tried to upgrade via helm and that fails because the images aren't available. This doesn't seem to be tied to pg version either, this image failed to pull: registry.developers.crunchydata.com/crunchydata/pgo-deployer:ubi8-4.7.10.

I'm watching operator releases and a few issues in this repo, and I had no clue this was coming.

[craigkerstiens]

How this ended up happening? https://www.postgresql.org/support/versioning/ PG13 is still in production cycle...

Apologies for the confusion. Crunchy Postgres for Kubernetes v4.7 was initially release almost two years ago. With the release of Crunchy Postgres for Kubernetes v5 (and subsequently v5.1, v5.2 and v5.3) we are no longer making v4.7 available through the Crunchy Data Developer Program - as the combination of new features and underlying Kubernetes version support makes the v5 series a better place for deployments to start. To the extent you need long term support for Crunchy Postgres for Kubernetes v4.7, we have some option there for paying customers. We're continuing to explore how we better communicate these changes with the PGO community. We do not want to leave you in a bad place so please contact us at info@crunchydata.com so we can help sort through how best we can assist.

[keithlayne]

@craigkerstiens Would be great to understand what "developer portal images" means. Installing v5 of the operator via the registry (predictably) installs images from that registry. I'm probably reading this wrong, but I'm reading it as "when you install pgo, you are going to be using images not intended for production."

It kinda seems like you just deleted all the 4.x images from your image repo, and you kinda acknowledge that the communication wasn't great. Hard to feel really good about this from my perspective.

[craigkerstiens]

I mean, 4.7.10 was released yesterday, I just tried to upgrade via helm and that fails because the images aren't available. This doesn't seem to be tied to pg version either, this image failed to pull: registry.developers.crunchydata.com/crunchydata/pgo-deployer:ubi8-4.7.10.

I'm watching operator releases and a few issues in this repo, and I had no clue this was coming.

The ideal place to follow for updates on releases, end of builds, etc. is on our mailing list. Would recommend joining it at https://www.crunchydata.com/developers/download-postgres/containers.

4.7.10 is intended more as a LTS for paying customers, but has been in feature freeze for some time.

As mentioned elsewhere, v5 and the subsequent versions over the last two years have a number of improvements and make keeping current with modern K8s versions much easier thus we recommend getting to those more modern versions.

[keithlayne]

Thank you for clarifying. v5 has been a todo for a long time for us, but for other reasons that I can't address today, I'm stuck on an EOLed k8s version that's not compatible with pgo v5 (at least the latest).

Honestly the best option for me now seems to be to take the images I depend on and push them to a registry I control, from the one of the nodes that has already pulled them. This feels vaguely dirty.

[bauerjs1]

Sorry but I am still confused. We are using PGO v5 with the recommended images from the installation instructions, but pulling those still fails. If that is desired behavior, this should be made clear in the documentation and those images should be removed from the official examples.

See also: https://github.com/CrunchyData/postgres-operator-examples/issues/190

Are there any images that can be used for PG13 at this moment?

I think, from a user's perspective, the best way to avoid such scenarios in the future is to use a local proxy cache for the images.

[idanl21]

Hey, i managed to upload your images to my private registry and changed the ccp_image and pog_image env so i can create clusters and backups. But when im trying to create a new cluster from existing cluster the bootstrap pod is still using the old registry, i tried to look everywhere for that image build declaration , but cannot find it so i cannot restore a new cluster.

How do you build the image address for the bootstrap pod ? and where i can change it? @craigkerstiens

[stemid]

@craigkerstiens can you enable the 13.10 image temporarily so we can upgrade our clusters? I had not received any word of this. I will try to watch the dev list from now on. But right now my cluster is dead because I can't upgrade to 14 with an unhealthy cluster.

And btw @craigkerstiens when I try to join the developer list I just get an error saying "Nice try robot.".

[andrewlecuyer]

Hi @stemid, sorry to hear that you are stuck! We definitely do not want to leave you in a bad place.

Therefore, please contact us at info@crunchydata.com to help sort through how we can best assist you with upgrading your various PostgreSQL clusters.

[haf]

What @stemid said—the deprecation of removal of the images was unexpected. I've written to info@crunchydata.com now.

[stemid]

What @stemid said—the deprecation of removal of the images was unexpected. I've written to info@crunchydata.com now.

@andrewlecuyer @haf I managed to resolve the situation. I did a docker save of the old 13 image from an existing cluster, then docker load into my own repo. That way I could bring the cluster back to healthy status and at least upgrade it. During the upgrade I actually found a bug in the pgupgrade process that I reported in your issues.

I will continue testing your Development version and keep a close eye on the development mailing list from now on. You make a great product and I use it for hosting non-profit organisations.

[haf]

@stemid They haven't yet written back... Would it be possible for you to send them to me: henrik at causiq dot com? How did you go about renaming them? Because I presume you loaded the new image as the image to be used in the crunchy controller and let the controller create the stateful sets itself?

[captainjapeng]

Hi! I've encountered the same issue as well... For those who're looking for a PG v13 image, you may use this one.

ghcr.io/captainjapeng/crunchy-postgres:ubi8-13.9-2

[neddodiallo]

Hi @captainjapeng can you help me with this image ghcr.io/captainjapeng/crunchy-postgres:ubi8-13.9-2.

[neddodiallo]

@craigkerstiens Would be great to understand what "developer portal images" means. Installing v5 of the operator via the registry (predictably) installs images from that registry. I'm probably reading this wrong, but I'm reading it as "when you install pgo, you are going to be using images not intended for production."

It kinda seems like you just deleted all the 4.x images from your image repo, and you kinda acknowledge that the communication wasn't great. Hard to feel really good about this from my perspective.

Do you have solution?

[neddodiallo]

What @stemid said—the deprecation of removal of the images was unexpected. I've written to info@crunchydata.com now.

@andrewlecuyer @haf I managed to resolve the situation. I did a docker save of the old 13 image from an existing cluster, then docker load into my own repo. That way I could bring the cluster back to healthy status and at least upgrade it. During the upgrade I actually found a bug in the pgupgrade process that I reported in your issues.

I will continue testing your Development version and keep a close eye on the development mailing list from now on. You make a great product and I use it for hosting non-profit organisations.

Can you please share the image?

[luatnd]

I recently can setup crunchy on my local machine, but today I got this error on my VPS:

│   Type     Reason     Age                   From               Message                                  │
│   ----     ------     ----                  ----               -------                                  │
│   Normal   Scheduled  4m56s                 default-scheduler  Successfully assigned postgres-operator/ │
│ pgo-5cbcf7c466-4tmnr to hel1-2c4g                                                                       │
│   Normal   Pulling    117s (x4 over 4m56s)  kubelet            Pulling image "registry.developers.crunc │
│ hydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0"                                                  │
│   Warning  Failed     87s (x4 over 4m26s)   kubelet            Failed to pull image "registry.developer │
│ s.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0": rpc error: code = DeadlineExceeded desc  │
│ = failed to pull and unpack image "registry.developers.crunchydata.com/crunchydata/postgres-operator:ub │
│ i8-5.4.2-0": failed to resolve reference "registry.developers.crunchydata.com/crunchydata/postgres-oper │
│ ator:ubi8-5.4.2-0": failed to do request: Head "https://registry.developers.crunchydata.com/v2/crunchyd │
│ ata/postgres-operator/manifests/ubi8-5.4.2-0": dial tcp 34.170.221.86:443: i/o timeout                  │
│   Warning  Failed     87s (x4 over 4m26s)   kubelet            Error: ErrImagePull                      │
│   Warning  Failed     75s (x6 over 4m25s)   kubelet            Error: ImagePullBackOff                  │
│   Normal   BackOff    63s (x7 over 4m25s)   kubelet            Back-off pulling image "registry.develop │
│ ers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0"       

As I checked 5.4.2 is the latest version at this time. I hosted with k3s on hetzner arm64, can you suggest me the approach to fix this?

[andrewlecuyer]

@luatnd we are not seeing any issues pulling the image:

 % podman pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0
Trying to pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0...
Getting image source signatures
Copying blob sha256:ba5bedb8c7e8b3d862ba4d517d65bd544c01cf5549d58d9e4efb87bed7cfef74
Copying blob sha256:96aaf4ff1a6663ef5e5081b8f83265acd80a5a4e725bfac21bf247b1ce6bb252
Copying blob sha256:8b44edf5cc0f19a3d65b8145b40e7f3205de7a90efb7a13572ec376e93b0ac96
Copying config sha256:f40fc1e57fcb014451e7c12a5abbd5181aa4981b82a75937b4b4faeb87209835
Writing manifest to image destination
f40fc1e57fcb014451e7c12a5abbd5181aa4981b82a75937b4b4faeb87209835

Therefore, the issue does not appear to be with the registry.

Is the issue still occurring? If so, perhaps there is a network issue on your end?

[luatnd]

@luatnd we are not seeing any issues pulling the image:

 % podman pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0
Trying to pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0...
Getting image source signatures
Copying blob sha256:ba5bedb8c7e8b3d862ba4d517d65bd544c01cf5549d58d9e4efb87bed7cfef74
Copying blob sha256:96aaf4ff1a6663ef5e5081b8f83265acd80a5a4e725bfac21bf247b1ce6bb252
Copying blob sha256:8b44edf5cc0f19a3d65b8145b40e7f3205de7a90efb7a13572ec376e93b0ac96
Copying config sha256:f40fc1e57fcb014451e7c12a5abbd5181aa4981b82a75937b4b4faeb87209835
Writing manifest to image destination
f40fc1e57fcb014451e7c12a5abbd5181aa4981b82a75937b4b4faeb87209835

Therefore, the issue does not appear to be with the registry.

Is the issue still occurring? If so, perhaps there is a network issue on your end?

Thank you so much for helping me, tried on other cloud providers and my local machine, both can pull the image, but hetzner cannot! I'll work with my cloud provider on this issue.

Btw, can you confirm that registry.developers.crunchydata.com does not put some IP from hetzner into the blacklist?

[luatnd]

Cloud provider said that they don't block any address, but I need to migrate that VPS to the new one to get a new IP, then it work! :(

[bap-dev]

Hi, we are experiencing this issue with a newer version: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0

in kuberenetes: Failed to pull image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": rpc error: code = NotFound desc = failed to pull and unpack image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": failed to resolve reference "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11 -0: not found

with curl -L this occurs: { "errors":[{"code":"404","message":"This container version is no longer available from the Crunchy Data Developer Program. For information on accessing these containers, please contact info@crunchydata.com." }]} We haven't had any troubles up to this point.....

[neoighodaro]

Hi, we are experiencing this issue with a newer version: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0

in kuberenetes: Failed to pull image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": rpc error: code = NotFound desc = failed to pull and unpack image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": failed to resolve reference "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11 -0: not found

with curl -L this occurs: { "errors":[{"code":"404","message":"This container version is no longer available from the Crunchy Data Developer Program. For information on accessing these containers, please contact info@crunchydata.com." }]} We haven't had any troubles up to this point.....

Same issue

[bap-dev]

sorry for the double post: https://github.com/CrunchyData/postgres-operator/issues/3859