search

CrunchyData / postgres-operator

Production PostgreSQL for Kubernetes, from high availability Postgres clusters to full-scale database-as-a-service.
https://access.crunchydata.com/documentation/postgres-operator/v5/
Apache License 2.0
3.96k stars 593 forks source link

This container version is no longer available from the Crunchy Data Developer Program. #3601

Open idanl21 opened 1 year ago

idanl21 commented 1 year ago

Hey :) im using your Operator for Prod and i started to get that error on my cluster: Error response from daemon: unknown: This container version is no longer available from the Crunchy Data Developer Program. For information on accessing these containers, please contact info@crunchydata.com. Did you deleted those images? using images : registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:centos8-4.7.5 registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.3-4.7.0

CaptainKro commented 1 year ago

I have the same problem. I try to update the version of image with the last example and it's the same. Perhaps it's a registry problem.

All my production cluster is currently broken :-(

radepajic commented 1 year ago

The same applies to me. I have problem with pulling image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-13.8-1

craigkerstiens commented 1 year ago

Sorry to hear about this issue. There has been some discussion of this in past GitHub issues and communications through various channels, but the short answer is that Crunchy Data makes available the last two Postgres versions as part of the Postgres Developer program. We've sent out notifications on our developer mailing list regarding this and would recommend subscribing there.

Developer portal images are not intended for production use. But we very much do not want folks to be in a bad place. If you have feedback on how we could better improve communication would welcome that feedback. At the moment we're exploring how to better enhance and improve the community around this so hopefully that helps in the future.

If you can contact us at info@crunchydata.com we are happy to help you upgrade or if you need support for running in production would be happy to have a conversation around Crunchy Postgres for Kubernetes.

adriannieto-attechnest commented 1 year ago

How this ended up happening? https://www.postgresql.org/support/versioning/ PG13 is still in production cycle...

keithlayne commented 1 year ago

Developer portal images are not intended for production use.

@craigkerstiens - okay. Why do I have them in production? Is it because I don't have a paid subscription? I mean, 4.7.10 was released yesterday, I just tried to upgrade via helm and that fails because the images aren't available. This doesn't seem to be tied to pg version either, this image failed to pull: registry.developers.crunchydata.com/crunchydata/pgo-deployer:ubi8-4.7.10.

I'm watching operator releases and a few issues in this repo, and I had no clue this was coming.

craigkerstiens commented 1 year ago

How this ended up happening? https://www.postgresql.org/support/versioning/ PG13 is still in production cycle...

Apologies for the confusion. Crunchy Postgres for Kubernetes v4.7 was initially release almost two years ago. With the release of Crunchy Postgres for Kubernetes v5 (and subsequently v5.1, v5.2 and v5.3) we are no longer making v4.7 available through the Crunchy Data Developer Program - as the combination of new features and underlying Kubernetes version support makes the v5 series a better place for deployments to start. To the extent you need long term support for Crunchy Postgres for Kubernetes v4.7, we have some option there for paying customers. We're continuing to explore how we better communicate these changes with the PGO community. We do not want to leave you in a bad place so please contact us at info@crunchydata.com so we can help sort through how best we can assist.

keithlayne commented 1 year ago

@craigkerstiens Would be great to understand what "developer portal images" means. Installing v5 of the operator via the registry (predictably) installs images from that registry. I'm probably reading this wrong, but I'm reading it as "when you install pgo, you are going to be using images not intended for production."

It kinda seems like you just deleted all the 4.x images from your image repo, and you kinda acknowledge that the communication wasn't great. Hard to feel really good about this from my perspective.

craigkerstiens commented 1 year ago

I mean, 4.7.10 was released yesterday, I just tried to upgrade via helm and that fails because the images aren't available. This doesn't seem to be tied to pg version either, this image failed to pull: registry.developers.crunchydata.com/crunchydata/pgo-deployer:ubi8-4.7.10.

I'm watching operator releases and a few issues in this repo, and I had no clue this was coming.

The ideal place to follow for updates on releases, end of builds, etc. is on our mailing list. Would recommend joining it at https://www.crunchydata.com/developers/download-postgres/containers.

4.7.10 is intended more as a LTS for paying customers, but has been in feature freeze for some time.

As mentioned elsewhere, v5 and the subsequent versions over the last two years have a number of improvements and make keeping current with modern K8s versions much easier thus we recommend getting to those more modern versions.

keithlayne commented 1 year ago

Thank you for clarifying. v5 has been a todo for a long time for us, but for other reasons that I can't address today, I'm stuck on an EOLed k8s version that's not compatible with pgo v5 (at least the latest).

Honestly the best option for me now seems to be to take the images I depend on and push them to a registry I control, from the one of the nodes that has already pulled them. This feels vaguely dirty.

bauerjs1 commented 1 year ago

Sorry but I am still confused. We are using PGO v5 with the recommended images from the installation instructions, but pulling those still fails. If that is desired behavior, this should be made clear in the documentation and those images should be removed from the official examples.

See also: https://github.com/CrunchyData/postgres-operator-examples/issues/190

Are there any images that can be used for PG13 at this moment?

I think, from a user's perspective, the best way to avoid such scenarios in the future is to use a local proxy cache for the images.

idanl21 commented 1 year ago

Hey, i managed to upload your images to my private registry and changed the ccp_image and pog_image env so i can create clusters and backups. But when im trying to create a new cluster from existing cluster the bootstrap pod is still using the old registry, i tried to look everywhere for that image build declaration , but cannot find it so i cannot restore a new cluster.

How do you build the image address for the bootstrap pod ? and where i can change it? @craigkerstiens

stemid commented 1 year ago

@craigkerstiens can you enable the 13.10 image temporarily so we can upgrade our clusters? I had not received any word of this. I will try to watch the dev list from now on. But right now my cluster is dead because I can't upgrade to 14 with an unhealthy cluster.

And btw @craigkerstiens when I try to join the developer list I just get an error saying "Nice try robot.".

andrewlecuyer commented 1 year ago

Hi @stemid, sorry to hear that you are stuck! We definitely do not want to leave you in a bad place.

Therefore, please contact us at info@crunchydata.com to help sort through how we can best assist you with upgrading your various PostgreSQL clusters.

haf commented 1 year ago

What @stemid said—the deprecation of removal of the images was unexpected. I've written to info@crunchydata.com now.

stemid commented 1 year ago

What @stemid said—the deprecation of removal of the images was unexpected. I've written to info@crunchydata.com now.

@andrewlecuyer @haf I managed to resolve the situation. I did a docker save of the old 13 image from an existing cluster, then docker load into my own repo. That way I could bring the cluster back to healthy status and at least upgrade it. During the upgrade I actually found a bug in the pgupgrade process that I reported in your issues.

I will continue testing your Development version and keep a close eye on the development mailing list from now on. You make a great product and I use it for hosting non-profit organisations.

haf commented 1 year ago

@stemid They haven't yet written back... Would it be possible for you to send them to me: henrik at causiq dot com? How did you go about renaming them? Because I presume you loaded the new image as the image to be used in the crunchy controller and let the controller create the stateful sets itself?

captainjapeng commented 1 year ago

Hi! I've encountered the same issue as well... For those who're looking for a PG v13 image, you may use this one.

ghcr.io/captainjapeng/crunchy-postgres:ubi8-13.9-2

neddodiallo commented 1 year ago

Hi @captainjapeng can you help me with this image ghcr.io/captainjapeng/crunchy-postgres:ubi8-13.9-2.

neddodiallo commented 1 year ago

@craigkerstiens Would be great to understand what "developer portal images" means. Installing v5 of the operator via the registry (predictably) installs images from that registry. I'm probably reading this wrong, but I'm reading it as "when you install pgo, you are going to be using images not intended for production."

It kinda seems like you just deleted all the 4.x images from your image repo, and you kinda acknowledge that the communication wasn't great. Hard to feel really good about this from my perspective.

Do you have solution?

neddodiallo commented 1 year ago

What @stemid said—the deprecation of removal of the images was unexpected. I've written to info@crunchydata.com now.

@andrewlecuyer @haf I managed to resolve the situation. I did a docker save of the old 13 image from an existing cluster, then docker load into my own repo. That way I could bring the cluster back to healthy status and at least upgrade it. During the upgrade I actually found a bug in the pgupgrade process that I reported in your issues.

I will continue testing your Development version and keep a close eye on the development mailing list from now on. You make a great product and I use it for hosting non-profit organisations.

Can you please share the image?

luatnd commented 1 year ago

I recently can setup crunchy on my local machine, but today I got this error on my VPS:

│   Type     Reason     Age                   From               Message                                  │
│   ----     ------     ----                  ----               -------                                  │
│   Normal   Scheduled  4m56s                 default-scheduler  Successfully assigned postgres-operator/ │
│ pgo-5cbcf7c466-4tmnr to hel1-2c4g                                                                       │
│   Normal   Pulling    117s (x4 over 4m56s)  kubelet            Pulling image "registry.developers.crunc │
│ hydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0"                                                  │
│   Warning  Failed     87s (x4 over 4m26s)   kubelet            Failed to pull image "registry.developer │
│ s.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0": rpc error: code = DeadlineExceeded desc  │
│ = failed to pull and unpack image "registry.developers.crunchydata.com/crunchydata/postgres-operator:ub │
│ i8-5.4.2-0": failed to resolve reference "registry.developers.crunchydata.com/crunchydata/postgres-oper │
│ ator:ubi8-5.4.2-0": failed to do request: Head "https://registry.developers.crunchydata.com/v2/crunchyd │
│ ata/postgres-operator/manifests/ubi8-5.4.2-0": dial tcp 34.170.221.86:443: i/o timeout                  │
│   Warning  Failed     87s (x4 over 4m26s)   kubelet            Error: ErrImagePull                      │
│   Warning  Failed     75s (x6 over 4m25s)   kubelet            Error: ImagePullBackOff                  │
│   Normal   BackOff    63s (x7 over 4m25s)   kubelet            Back-off pulling image "registry.develop │
│ ers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0"

As I checked 5.4.2 is the latest version at this time. I hosted with k3s on hetzner arm64, can you suggest me the approach to fix this?

andrewlecuyer commented 1 year ago

@luatnd we are not seeing any issues pulling the image:

 % podman pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0
Trying to pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0...
Getting image source signatures
Copying blob sha256:ba5bedb8c7e8b3d862ba4d517d65bd544c01cf5549d58d9e4efb87bed7cfef74
Copying blob sha256:96aaf4ff1a6663ef5e5081b8f83265acd80a5a4e725bfac21bf247b1ce6bb252
Copying blob sha256:8b44edf5cc0f19a3d65b8145b40e7f3205de7a90efb7a13572ec376e93b0ac96
Copying config sha256:f40fc1e57fcb014451e7c12a5abbd5181aa4981b82a75937b4b4faeb87209835
Writing manifest to image destination
f40fc1e57fcb014451e7c12a5abbd5181aa4981b82a75937b4b4faeb87209835

Therefore, the issue does not appear to be with the registry.

Is the issue still occurring? If so, perhaps there is a network issue on your end?

luatnd commented 1 year ago

@luatnd we are not seeing any issues pulling the image:

 % podman pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0
Trying to pull registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.4.2-0...
Getting image source signatures
Copying blob sha256:ba5bedb8c7e8b3d862ba4d517d65bd544c01cf5549d58d9e4efb87bed7cfef74
Copying blob sha256:96aaf4ff1a6663ef5e5081b8f83265acd80a5a4e725bfac21bf247b1ce6bb252
Copying blob sha256:8b44edf5cc0f19a3d65b8145b40e7f3205de7a90efb7a13572ec376e93b0ac96
Copying config sha256:f40fc1e57fcb014451e7c12a5abbd5181aa4981b82a75937b4b4faeb87209835
Writing manifest to image destination
f40fc1e57fcb014451e7c12a5abbd5181aa4981b82a75937b4b4faeb87209835

Therefore, the issue does not appear to be with the registry.

Is the issue still occurring? If so, perhaps there is a network issue on your end?

Thank you so much for helping me, tried on other cloud providers and my local machine, both can pull the image, but hetzner cannot! I'll work with my cloud provider on this issue.

Btw, can you confirm that registry.developers.crunchydata.com does not put some IP from hetzner into the blacklist?

luatnd commented 1 year ago

Cloud provider said that they don't block any address, but I need to migrate that VPS to the new one to get a new IP, then it work! :(

bap-dev commented 9 months ago

Hi, we are experiencing this issue with a newer version: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0

in kuberenetes: Failed to pull image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": rpc error: code = NotFound desc = failed to pull and unpack image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": failed to resolve reference "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11 -0: not found

with curl -L this occurs: { "errors":[{"code":"404","message":"This container version is no longer available from the Crunchy Data Developer Program. For information on accessing these containers, please contact info@crunchydata.com." }]} We haven't had any troubles up to this point.....

neoighodaro commented 9 months ago

Hi, we are experiencing this issue with a newer version: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0

in kuberenetes: Failed to pull image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": rpc error: code = NotFound desc = failed to pull and unpack image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": failed to resolve reference "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11 -0: not found

with curl -L this occurs: { "errors":[{"code":"404","message":"This container version is no longer available from the Crunchy Data Developer Program. For information on accessing these containers, please contact info@crunchydata.com." }]} We haven't had any troubles up to this point.....

Same issue

bap-dev commented 9 months ago

sorry for the double post: https://github.com/CrunchyData/postgres-operator/issues/3859

muhammadyusuf-kurbonov commented 1 week ago

ADD WARNING OR ERROR ON DOCS!!! IF IT IS NOT AVAILABLE!!! Very bad implementation of such solution. I spent my time waiting for pulling and setting up cluster! Thanx

andrewlecuyer commented 1 week ago

@muhammadyusuf-kurbonov sorry for the confusion. Please see the Supported Platforms page in the documentation for information about the versions of CPK and Postgres that are currently supported via the Crunchy Developer Program:

https://access.crunchydata.com/documentation/postgres-operator/latest/overview/supported-platforms

For the best experience and community support, we recommend using to a version that is currently active/supported.

And if you think the pull error you're seeing is an error (e.g. according to the versions that are available per the Supported Platforms page), please let me know and I'm happy to take a closer look. Based on your message, I wasn't able to determine the specific version of CPK, Postgres, etc. you were attempting to use.