Pub/Sub emulator fails with Java JRE missing

atombender commented 1 month ago

Looks like a regression of #290.

$ docker run -it gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators gcloud beta emulators pubsub start --project=blah --host-port=0.0.0.0:8085
ERROR: (gcloud.beta.emulators.pubsub.start) To use the Google Cloud Pub/Sub emulator, a Java 7+ JRE must be installed and on your system PATH

P<robably caused by #482.

aleclerc-cio commented 1 month ago

Yes can confirm, we hit this as well.

aabdelli13 commented 1 month ago

Same for me

millotp commented 1 month ago

We have the same issue, but version 496.0.0 was working last week, it seems that the tag was overwritten 2h ago

Techwolf12 commented 1 month ago

Also running into it here

iliuta commented 1 month ago

Same for me. I used the 496.0.0-emulators image last week without any problem. It looks like someone pushed a bad thing on the image tag a few hours ago. Nothing works anymore.

ls-urs-keller commented 1 month ago

we reverted to a @sha256:<tag> E.g. this one: https://console.cloud.google.com/artifacts/docker/google.com:cloudsdktool/us/gcr.io/google-cloud-cli/sha256:98634552ada78aac1fea3a19aa2e3f245f764b71a9af11c7d6202a89d1e72212

docker run -ti gcr.io/google.com/cloudsdktool/google-cloud-cli@sha256:98634552ada78aac1fea3a19aa2e3f245f764b71a9af11c7d6202a89d1e72212 gcloud beta emulators pubsub start --project=emulator --host-port 0.0.0.0:8085 --verbosity debug

See here for other images: https://console.cloud.google.com/artifacts/docker/google.com:cloudsdktool/us/gcr.io/google-cloud-cli

Jarvvski commented 1 month ago

Made a quick PR to add it back in - not sure if that fully fixes it

https://github.com/GoogleCloudPlatform/cloud-sdk-docker/pull/487

anindyatahsin commented 1 month ago

Since the base image was updated to debian-12, openjdk-11 would not work anymore. We are adding the open-jdk-17 and a new release of the images are coming soon.

iliuta commented 1 month ago

I would need to understand though. Why the tag 496.0.0-emulators was overwritten? Is that a common practice here or am I missing something? In my imagination, a tag is by definition immutable. Should I expect that today I pull x.y.z version of an image it works one way and tomorrow if I pull the same version, not only it hasn't the same content but also it works differently?

Jarvvski commented 1 month ago

Since the base image was updated to debian-12, openjdk-11 would not work anymore. We are adding the open-jdk-17 and a new release of the images are coming soon.

That's fine.

Ultimately, this is googles repo, and its your emulators. However, I just want to be sure it's made aware;

to fix this, you just made a commit straight to master
overwrote the original 496.0.0-emulators version tag so even if we pin versions, we would be affected
also changed the java version (again, your emulators - do as you please)

Just flagging that this is a little 'out there' as far as process goes. In my companies specific use case, this bug broke all our PRs for atleast an hour - because no PR could pass CI due to emulators failing. Needing an engineer to go check around, and eventually landing at this issue

Can someone at Google confirm that steps are going to be taken to prevent image overriding in the future?

mathieu-lemay commented 1 month ago

Since the base image was updated to debian-12, openjdk-11 would not work anymore. We are adding the open-jdk-17 and a new release of the images are coming soon.

That's fine.

Ultimately, this is googles repo, and its your emulators. However, I just want to be sure it's made aware;
* to fix this, you just [_made a commit straight to master_](https://github.com/GoogleCloudPlatform/cloud-sdk-docker/commit/8c72af1b719c3a708520e320f068224baa3f89fd)

* overwrote the original `496.0.0-emulators` version tag so even if we pin versions, we would be affected

* also changed the java version (again, your emulators - do as you please)
Just flagging that this is a little 'out there' as far as process goes. In my companies specific use case, this bug broke all our PRs for atleast an hour - because no PR could pass CI due to emulators failing. Needing an engineer to go check around, and eventually landing at this issue

Can someone at Google confirm that steps are going to be taken to prevent image overriding in the future?

The fact that they pushed a non-working image shows they don't even have one automated test since this would have been caught immediately.

anindyatahsin commented 1 month ago

The emulator issue should be resolved on :496.0.0-emulators, :497.0.0-emulators and :emulators tags.

The 496.0.0-emulators image is using openjdk-11-jre-headless while version 497.0.0-emulators is using the debian-12 image with openjdk-17-jre-headless.

For production environments, we recommend using the image with a dated version tag (i.e.- :496.0.0-emulators-20241014) or a specific image hash (gcr.io/google.com/cloudsdktool/google-cloud-cli@sha256:98634552ada78aac1fea3a19aa2e3f245f764b71a9af11c7d6202a89d1e72212) for the docker images. The gcloud cli docker images are released everyday to deal with base image security vulnerabilities.

Closing this thread now. Feel free to reopen if the issue is not resolved for you.

atombender commented 1 month ago

For production environments, we recommend using the image with a dated version tag (i.e.- :496.0.0-emulators-20241014)

Not sure if you realize that this is undocumented. It's not mentioned in the official docs, nor in the readme. In fact, the official recommendation is to use a versioned image tag without a date.

It's also very unusual to overwrite image tags. Tags should be immutable so as to not suddenly break production environments such as CI builds. The docs also make no mention of the fact that a tag will be overwritten. My recommendation: If you need to apply security fixes, create a patch version. How else is anyone going to know that you updated the image?

Ironically, the docs say:

To avoid possible breaking changes, we recommend that your production environment use a versioned image, such as :489.0.0-stable, that you update periodically.

…which does not seem to be true.

iliuta commented 1 month ago

It's also very unusual to overwrite image tags. Tags should be immutable so as to not suddenly break production environments such as CI builds.

@atombender Totally agree with you. What are the patch numbers for? I lost like 3 hours yesterday figuring out why my CICD pipeline stopped working because someone thought it's OK to overwrite a tag. One could argue with reason that this is open source and I'm not obliged to use it. But even in open source there are good practices and bad practices. And these are "official" Google images.

anindyatahsin commented 1 month ago

@atombender Thanks for pointing that out. We have already updated the recommendation for using the docker images in prod environment.

We are working on a more streamlined approach of managing the tags so that this does not happen in the future.

ferha-google commented 1 month ago

We acknowledge that the recent changes in our image build process caused frustrating breakages. Specifically:

breaking the pub/sub emulator when migrating to debian-12
moving versioned tags, on a major base image update.

anindyatahsin has addressed the immediate issue.

We are also making a few changes immediately:

version tags will be static. So if we rebuild a image within the same gcloud VERSION, then the new image will have a new tag VERSION-tag-yyyymmdd, while the tag VERSION-tag will stay with original image.
The official documentation will clearly guide what tags to use in production environments.

We apologize for the inconvenience, and as we continue to make improvements to the images your feedback is truly appreciated!

atombender commented 1 month ago

Much appreciated!

GoogleCloudPlatform / cloud-sdk-docker

Pub/Sub emulator fails with Java JRE missing #486