OpenJDK 8u212 and 11.0.3 image quality concerns

[vielmetti]

I'm reading

https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-May/009330.html

entitled "Mystery meat OpenJDK builds strike again", in which it is noted that the openjdk:8 and openjdk:11 versions were observed to be not release versions but instead pre-release versions, originating from Debian and not reflecting the actual final releases of those versions.

[tianon]

@robilad we could definitely use some "official" advice here if you could spare any or refer us to any other authoritative folks willing to help out :sweat_smile:

If these builds are not representing OpenJDK well, I'd like to remove or more preferably replace them, but I want to make sure that we don't just trade one sideways situation for another.

To be totally clear, this sort of oddness with the Debian builds is exactly why for 12+ we've switched to be based entirely on the builds officially recommended by the OpenJDK project (https://jdk.java.net/, https://github.com/docker-library/openjdk/pull/235). There's an ongoing discussion in https://github.com/docker-library/openjdk/issues/302 to add back Debian-based images, but they'd still be based on those "official" builds (not Debian's packages).

I also think it's reasonable to point out https://github.com/docker-library/official-images/pull/5710 here, which is an initiative from the AdoptOpenJDK project to add official AdoptOpenJDK images maintained directly by that community. :+1:

Perhaps if that were available as a viable alternative for users who need these older versions, we could more seriously consider simply dropping any builds older than OpenJDK 12 (given that 12+ is the only place we've got officially-recommended binaries from OpenJDK officially)?

[vielmetti]

attn @giltene

[tianon]

I think it also makes sense to point out https://hub.docker.com/_/amazoncorretto and https://hub.docker.com/_/ibmjava as viable official-images alternatives maintained by their respective upstreams, although with the caveat that the latter is likely to be superseded by AdoptOpenJDK.

[giltene]

@tianon I'd be happy to provide constructive advice and direction on how the Official Docker images for OpenJDK might be able to regain credibility over time.

The first step in doing that is to avoid "making excuses for" the severity of the current situation (and the identical situation that occurred in 2014-2015, referred to here: https://github.com/docker-library/openjdk/commit/00a9c5c080f2a5fd1510bc0716db7afe06cbd017#commitcomment-8032128 ). It happened. That sucks. Now let's work on fixing things so it doesn't happen again.

By now, it should be all too apparent that the debian builds that have been used by the official docker images for OpenJDK to satisfy your users' needs for a current, updated OpenJDK are/were the wrong tool for the job. Stable debian builds are often dramatically out of date (e.g. the current stable builds are at least 9 months behind on bug fixes and security updates), and the seemingly-more-up-to-date builds found in various non-default debian repos (unstable, stretch-backports, etc.) are regularly mislabeled in a way that makes it "hard" to differentiate bad (unreleased, built from top-of-tree, and full of bugs and security holes not expected in the version that they report being) from good builds.

Since using debian builds has caused this serious issue across multiple versions and significant time periods (at the very least for 8u40, 8u212, and 11.0.3, but may have happened multiple other times in between for all I know), repeating that mistake a third time is the obvious thing we need to avoid if we intend to rebuild confidence in the trustworthiness of official docker images for OpenJDK. The only reasonable remedy is (obviously) to stop using debian builds in official docker images for openjdk, and to find an alternative, reputable and reliable source for JDK and JRE builds for use in your official images.

This was suggested before (see link above), but not acted on at the time. I believe this was driven by misconceptions that time, distance, and the severity of the current situation have hopefully helped clarify by now. At the core (and seemingly the reason you did not switch to Zulu back then and prevent the problem from happening again) was the reaction to the proposed solution having a "brand", and the notion that if something's "name" is spelled differently than "openjdk" it is not a build of openjdk, but a "compatible variant" that is somehow suspect. In fact (and with 4+ year track record since then), the opposite is demonstrably true. With the appearance of various OpenJDK builds over the years, you will note that ALL reputable builds of OpenJDK that are distributed outside of an OS's native repo have "name" brand labels, for the simple reason that the OpenJDK trademark fair use rules demand it (and reputable sources tend to try to adhere to trademark rules). They might say things like "Zulu is a build of OpenJDK", but will carefully avoid using "OpenJDK" in an actual name. The "brand name" of the thing you will want to download may be "Zulu", or "IcedTea", or "Corretto", or "Liberica", or "AdoptOpenJDK", but it will never be plain "OpenJDK".

If the OS distro that you base your images on happens to distribute high quality, timely updates of OpenJDK on a regular and reliable basis, you may be ok using the build-in repos to get the JDKs for these images. But if not, and you end up needing a reliable source for JDKs and JREs to include in your Official Docker images for OpenJDK for some OSs (as you clearly needed to do to get openjdk:8 up to date), you need an alternative and reliable source for JDK and JRE binaries.

I would obviously recommend Zulu's 100% free, 100% OSS, reliably delivered builds for those OSs for this purpose. By now, the fit and quality of Zulu for this purpose should be well-established. Zulu's long track record makes it clear where good 100% free, 100% OSS, never-mislabeled, always tested builds of OpenJDK can be reliably found on the internet.

Recently, some other potential sources for OpenJDK builds have appeared, including several newcomers with good potential for establishing a reputation by showing consistent delivery. At this point in time, however, when it comes to both breadth (in platforms, packages, availability via public repos, etc.) and length of actual demonstrated delivery (number of good quality quarterly builds delivered for free, with an established track record of NOT delivering bad material), Zulu is unmatched as a reliable and reliably free source of OpenJDK builds.

Since Zulu is available (amongst many other delivery forms) through a deb repo, the following example shows the straightforward changes that would allow you to cleanly transition your current openjdk:8 Dockerfile to one that delivers a good (actually released) 8u212:

18a19,20
> RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0xB1998361219BD9C9
> RUN echo "deb http://repos.azulsystems.com/debian stable  main" >> /etc/apt/sources.list.d/zulu.list
39c41
< RUN ln -svT "/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)" /docker-java-home
---
> RUN ln -svT "/usr/lib/jvm/zulu-8-$(dpkg --print-architecture)" /docker-java-home
43c45
< ENV JAVA_DEBIAN_VERSION 8u212-b01-1~deb9u1
---
> ENV JAVA_ZULU_VERSION 8.38.0.13
54c56
<       openjdk-8-jdk="$JAVA_DEBIAN_VERSION" \
---
>       zulu-8="$JAVA_ZULU_VERSION" \

[vielmetti]

Noting https://github.com/geonetwork/docker-geonetwork/issues/23 as related.

[adinn]

If you are looking for Linux binary releases for the jdk8u and jdk11u projects you can find both: latest official releases and latest early access versions for the next upcoming release at the following URL:

https://adoptopenjdk.net/upstream.html

As the blurb says:

"These binaries are built by Red Hat on behalf of the OpenJDK jdk8u and jdk11u projects. The binaries are created from the unmodified source code at OpenJDK. Support is not provided."

i.e. these are vanilla binaries that 1) 100% reflect the shared, public repo code and 2) should work on all current, mainstream Linux releases (Debian included).

Note: that last sentence about 'support' simply means you don't have a phone to ring or an email to post to if you want help using them. If you want either of those you need to pay someone like Red Hat, Azul, AdoptOpenJDK etc.

Note: It does not mean these binaries are a one-off thrown over the wall release. The EA binaries will be updated regularly as new fixes are added to the jdk8u and jdk11u projects, The official binaries will be updated with security patches and critical fixes every 3 months when the security embargo is lifted.

Note: although it is not relevant in this forum there are also Windows releases at the above URL provided under the same terms.

[theRealAph]

"These binaries are built by Red Hat on behalf of the OpenJDK jdk8u and jdk11u projects. The binaries are created from the unmodified source code at OpenJDK. Support is not provided."

i.e. these are vanilla binaries that 1) 100% reflect the shared, public repo code and 2) should work on all current, mainstream Linux releases (Debian included).

Note: that last sentence about 'support' simply means you don't have a phone to ring or an email to post to if you want help using them. If you want either of those you need to pay someone like Red Hat, Azul, AdoptOpenJDK etc.

Note: It does not mean these binaries are a one-off thrown over the wall release. The EA binaries will be updated regularly as new fixes are added to the jdk8u and jdk11u projects, The official binaries will be updated with security patches and critical fixes every 3 months when the security embargo is lifted.

Indeed. I should do something about getting that wording fixed.

[tianon]

It seems as though there's been some misunderstanding, so I want to start with a bit of a level set.

1. https://github.com/docker-library/openjdk/pull/235 was done in collaboration with @robilad as an upstream representative to add new variants which consume the official OpenJDK artifacts from https://jdk.java.net/ which are linked from https://openjdk.java.net/ (specifically because we knew the Debian package situation was not great and we needed to do something)
2. they were new variants to avoid backwards compatibility concerns for existing image users
3. the fallout from our users on that action is in https://github.com/docker-library/openjdk/issues/302, where we're not discussing adding back Debian's packages, but are instead discussing taking those same official https://jdk.java.net/ artifacts and putting them in a Debian base in addition to what we have
4. for OpenJDK 12 and 13, we only have images which are consuming from https://jdk.java.net/, and have no plans to expand beyond that without an official upstream recommendation (preferably via an update to https://openjdk.java.net/ and/or https://jdk.java.net/)
5. https://twitter.com/tianon/status/1118670830626410496

   to be 110% clear, we would love and welcome the OpenJDK project to officially maintain the image instead of us, and would happily help coordinate that

Regarding Azul-based images, I think an Azul Zulu official image would be a great addition to the program, and I'd suggest taking a look at https://github.com/docker-library/official-images#contributing-to-the-standard-library -- I'm also happy to help do the actual legwork to help make that happen (whether that's writing Dockerfiles, making the PR over there, just advising, etc).

While I agree that Azul's Java packages are likely better than Debian, I don't think they're any more appropriate as an "official" OpenJDK image than Debian's are, so if we're going to switch from one unofficial package to another, I'd rather consider dropping 8 and 11 and suggesting users look at some of the alternatives that already exist in the official images today.

Regarding https://adoptopenjdk.net/upstream.html, I personally think that's a fantastic initiative and hope that it gains more traction. However, we asked @robilad about that and were given a similar answer to the concerns above -- essentially that they aren't artifacts officially released from the OpenJDK project, and thus don't seem appropriate for use here (which is why we're discussing the problems with the Debian-released artifacts in the first place, so IMO it's fair). I think this is something to work out within the OpenJDK project directly (and again, preferably get https://openjdk.java.net/ and/or https://jdk.java.net/ updated to note the outcome -- even noting the outcome publicly in the case of "no, these won't be considered official by the project" would be useful for users looking to know more about these artifacts IMO).

So here's what I think our next steps are (specifically in the context of the Docker image maintained here, https://hub.docker.com/_/openjdk -- hopefully someone involved in the OpenJDK community can take further community concerns and champion them over there):

1. ensure https://github.com/docker-library/official-images/pull/5710 continues to get the love and attention it needs to get an official AdoptOpenJDK image (for further Java-diversity in the official images)
2. possibly get a new Azul-based image we can recommend to users as well? :+1:
3. update the image description on https://hub.docker.com/_/openjdk to both point out where the 12 and 13 image artifacts come from and to officially note the deprecation of the 8 and 11 artifacts, officially suggesting that users of those versions consider alternatives (which we'll link directly to) such as adoptopenjdk, amazoncoretto, ibmjava (and hopefully Azul Zulu?)

[adinn]

I'm not really sure what you (or, perhaps I mean Dalibor Topic of Oracle -- aka @robilad) mean by this comment:

"Regarding https://adoptopenjdk.net/upstream.html ... we asked @robilad about that and were given a similar answer to the concerns above -- essentially that they aren't artifacts officially released from the OpenJDK project, and thus don't seem appropriate for use here."

I think you have been taking advice from the wrong person. Dalibor has pointed you at Oracle-built binaries for jdk11u and jdk8u. They can in no way constitute an an 'official' release since Oracle are deliberately not participating in the OpenJDK updates project for jdk11u and jdk8u. Oracle explicitly devolved all responsibility for updates of those releases to non-Oracle members of the OpenJDK project. Oracle are continuing to provide their own proprietary versions of jdk11u and jdk8u based on their own private versions of these repositories but they are not co-operating with the project in maintaining the open source code base for jdk11u and jdk8u. Taking releases from them is like asking Microsoft to provide you with a Linux release.

By contrast, the releases at https://adoptopenjdk.net/upstream.html were built by Red Hat on behalf of the OpenJDK jdk11u and jdk8u update projects explicitly to provide 'official', upstream, vanilla releases independent from any releases provided by vendors. They have been built specially by Red Hat for this purpose as universal releases and,note well, are not the same as the releases Red Hat ships with its operating system releases even though they are based on the same sources.

Most importantly, I describe these releases as 'official' with a justification that Dalibor or Oracle cannot possibly pretend to. The jdk11u and jdk8u updates project lead (Andrew Haley, a Red Hat employee and OpenJDK governing board member at large i.e. elected representative of the OpenJDK community) explicitly asked for these releases to be produced and made available as updates projects releases. If Dalibor wants to use that word to describe the Oracle releases then I think you need to check which end he is talking out of.

[tianon]

Please keep the discussion civil and assume good faith.

As far as I am aware, https://openjdk.java.net/ is authoritative on OpenJDK -- is that not correct? Is there a better source we should be considering for authoritative information about OpenJDK, especially about the official status of https://adoptopenjdk.net/upstream.html?

[tianon]

Also, I want it to be clear that this was me paraphrasing @robilad's sentiment because I couldn't find the exact source -- he did not use those exact words, so attacking him is not only inappropriate, it is also unfair to him.

[adinn]

"As far as I am aware, https://openjdk.java.net/ is authoritative on OpenJDK -- is that not correct?"

No, not with regard to non-Oracle releases. Oracle will not let non-Oracle members of the OpenJDK project install releases on openjdk.java.net. So, the OpenJDK updates projects for jdk8u and jdk11u have been forced to distribute official binaries via another site. By contrast, Oracle have unilaterally placed their jdk11u and jdk8u releases on that site without any sanction from the updates project.

So, rather than ask those actually involved in the updates project whether their releases are 'official' you might perhaps want to ask Oracle (or at least the Oracle representative you talked to) why they consider it appropriate to be claiming that they are in a position to distribute official releases for an open source project they have quite explicitly refused to continue to be involved in.

"Is there a better source we should be considering for authoritative information about OpenJDK, especially about the official status of https://adoptopenjdk.net/upstream.html?"

You could perhaps take that up with Andrew Haley (aph@redhat.com) or, perhaps, Doug Lea (dl@cs.oswego.edu). Andrew is the current project lead for the jdk8u and jdk11u updates projects. He and Doug are the two people who were voted onto the OpenJDK governing board to represent the OpenJDK community.

I am sorry you considered my last response uncivil. I was trying very hard to be restrained in the face of what I (still) see as an extremely misguided, if not abusive, action. If I had posted what I wanted to say the air would have been much more blue.

Also, I should point out that this is not the first time that Dalibor has made public comments about the state of the updates project that have muddied the waters about the status of the various available releases, So I have a reason to be skeptical of his bona fides. I'd be very interested not only to hear Dalibor's exact words but also his explanation of why he thinks Oracle should be pushing out releases for the open project.

[theRealAph]

Please keep the discussion civil and assume good faith.

As far as I am aware, https://openjdk.java.net/ is authoritative on OpenJDK -- is that not correct? Is there a better source we should be considering for authoritative information about OpenJDK, especially about the official status of https://adoptopenjdk.net/upstream.html?

Yes, there is a better source: me. I am the OpenJDK 8 and 11 Updates OpenJDK project lead. If @robilad is saying that the builds at https://adoptopenjdk.net/upstream.html are not "official", I'd disagree. He is not involved in the upstream 8u and 11u projects.

While it is true that the OpenJDK Governing Board has not sanctioned those releases, they (or rather we, since I am a member) didn't sanction Oracle's OpenJDK releases either. As far as I am aware, the lead of an OpenJDK project is entitled to release binary builds, and there is clearly a need for them.

I'm happy to help with any further questions that you have.

[giltene]

@tianon I think you misunderstand the purpose of the reference implementation builds linked to in https://jdk.java.net/ . Those are certainly not the builds anyone will want to use if they expect updates after the first few months of a new Java version coming out. They stop being updated after the first 6 months.

The reference implementations pointed to there are useful for soon-after-initial-GA releases only. They are updated for 6 months after GA and then stopped. For the non-LTS "Feature releases" (e.g. 9, 10, 12, 13, ...), where the projects do not get further updates and move on to the next version, this may be fine. But for the long term releases (7, 8, 11, 17, ...), which are the versions that the vast majority of people will actually need to run, no updates will appear. Specifically: 11 LTS is not being updated there anymore (11.0.2 is the last one you'll see there, apparently), and the latest version of 8 linked to there is 8u40 (which is 4 years old).

The bottom line is that there is no "official place to get OpenJDK binaries" and there never has been, because OpenJDK is a source code project. Downstream builds are made and posted by various parties, in various places. Some are good, some not so good, and some are dangerous to use.

You have no choice but to choose a source you can trust. And (especially after the history that led us here), that trust should probably be based on a demonstrated history of delivering non-bad builds.

While some good (and blessed by the project lead) material can be found in other places, e.g. the recently created https://adoptopenjdk.net/upstream.html mentioned above, which covers a handful of setups (jdk only, tarball/zip only, glibc only for linux), I'll repeat my note that Zulu has been providing reliable builds of OpenJDK for several years now. The Zulu build matrix covers 100s of packages every quarter, and we test and certify each to make sure people can trust their provenance. These include a multitude of packages that are often needed by containers [jdks and jres, with and without openjfx, linux glibc, Alpine-native (musl libc), windows, mac, etc.], and delivery formats [tarballs, zips, dmgs, deb packages, rpm packages, msi packages, public deb and rpm repos, etc.].

You can keep looking around, and may end up picking some non-Zulu build that will satisfy your needs, but I urge you to make a choice and fix this quickly, as every additional day that passes by with bad (literally insecure) versions being delivered to users of the default official docker hub images for java 8 and 11 (e.g. what people get when they run "docker run -it --rm openjdk:8 java -version" right now) is causing lasting damage to the reputation of these images.

[theRealAph]

The bottom line is that there is no "official place to get OpenJDK binaries" and there never has been, because OpenJDK is a source code project. Downstream builds are made and posted by various parties, in various places. Some are good, some not so good, and some are dangerous to use.

I suppose we're quibbling about what the word "official" means, or can mean, in an open-source project. While the OpenJDK project itself doesn't bless binaries, I believe that it is in the interests of the OpenJDK Updates projects (and those of our users) to make reference binaries available. Whether these are "official" or not depends on your precise definition of the word: in its adjectival form the word means "Having the approval or authorization of an authority," and as the Project Lead I think I qualify. But in the end, as you say, it's up to the user to make a judgment.

Having said that, I agree with you on this:

I urge you to make a choice and fix this quickly, as every additional day that passes by with bad (literally insecure) versions being delivered to users of the default official docker hub images for java 8 and 11 (e.g. what people get when they run "docker run -it --rm openjdk:8 java -version" right now) is causing lasting damage to the reputation of these images.

Docker deserves a proper, compliance-tested OpenJDK image.

[giltene]

The bottom line is that there is no "official place to get OpenJDK binaries" and there never has been, because OpenJDK is a source code project. Downstream builds are made and posted by various parties, in various places. Some are good, some not so good, and some are dangerous to use.

I suppose we're quibbling about what the word "official" means, or can mean, in an open-source project. While the OpenJDK project itself doesn't bless binaries, I believe that it is in the interests of the OpenJDK Updates projects (and those of our users) to make reference binaries available. Whether these are "official" or not depends on your precise definition of the word: in its adjectival form the word means "Having the approval or authorization of an authority," and as the Project Lead I think I qualify. But in the end, as you say, it's up to the user to make a judgment.

My point is that there is no official build of OpenJDK updates. There never was and never will be. The project does not produce one, and Oracle does not produce one. And that's a good thing, because many many builds are needed.

A "blessed by the project lead" build is a fine thing, and I'm all for it. But I assume you don't intend to say "these good builds are more official than others" or "other builds of OpenJDK don't count, and this is the one you should use for OpenJDK if the thing you are building is called openjdk". The "blessed by project lead" build is just as downstream as any other. And it's a presumably good one.

If there was an official set of builds, then builds not on the list would be treated as "unofficial" by lay people, and (as the history of this very issue shows) lead people to resist them, or take on bad builds with misleading labels that somehow trick people into thinking they fall into the "official" category.

To give some concrete examples of the impracticality and ill-advisability of declaring official builds for a source project with so many packages and targets: If there were such a thing as an official place to get binary builds of OpenJDK 8 or 11 sources, where would the official build for a jre of OpenJDK 8u for Apline linux with musl libc (and not glibc) be? Or the official MSI build of a JRE for windows. Or the official deb package for OpenJDK 11.0.3 jdks? None of those can be found in any "official" OpenJDK-tagged place, and I doubt that we should constrain the set of builds people can use by deeming any of the good builds for those "unofficial".

[theRealAph]

My point is that there is no official build of OpenJDK updates. There never was and never will be. The project does not produce one, and Oracle does not produce one. And that's a good thing, because many many builds are needed.

I hear what you are saying. :-)

A "blessed by the project lead" build is a fine thing, and I'm all for it. But I assume you don't intend to say "these good builds are more official than others" or "other builds of OpenJDK don't count, and this is the one you should use for OpenJDK if the thing you are building is called openjdk".

No. I am, however, entitled to say that "This is a good build. I don't know about the others." And I will note that Red Hat doesn't have a stake in this: if you want support from Red Hat you have to use Red Hat binaries, which these aren't.

The "blessed by project lead" build is just as downstream as any other And it's a presumably good one.

I certainly hope it's a good one.

Some downstreams seem not to be competent to build OpenJDK, and are unwilling, unable, or do not have the necessary licences to test what they build. Therefore, there has to be a blessed build, if only to be able to say "Look, this is what a build should look like." Also, the project can request a user to test the blessed builds to make sure that the bug is reproducible there.

If there was an official set of builds, then builds not on the list would be treated as "unofficial" by lay people, and (as the history of this very issue shows) lead people to resist them, or take on bad builds with misleading labels that somehow trick people into thinking they fall into the "official" category.

OK.

To give some concrete examples of the impracticality and ill-advisability of declaring official builds for a source project with so many packages and targets: If there were such a thing as an official place to get binary builds of OpenJDK 8 or 11 sources, where would the official build for a jre of OpenJDK 8u for Apline linux with musl libc (and not glibc) be? Or the official MSI build of a JRE for windows. Or the official deb package for OpenJDK 11.0.3 jdks? None of those can be found in any "official" OpenJDK-tagged place, and I doubt that we should constrain the set of builds people can use by deeming any of the good builds for those "unofficial".

any more than you would want Oracle's choices of what they post to superce

I take your point.

There is a serious risk of fragmentation, where no-one knows quite what they're getting when they use an OpenJDK update. The "upstream" builds provide a baseline.

Please step into my shoes for a moment. Of course, every downstream packager and vendor would prefer their builds to be advertised with "java -version" in every Docker image. Who wouldn't? But as project lead I cannot push Red Hat builds because I have to be impartial. However, there is a screaming need for some known good binaries that I can recommend, and I have solved that problem in the only way I can think of, by producing unbranded vendor-neutral builds. In my opinion they would also provide an excellent basis for Docker images. But yes, Zulu builds would be fine too.

[giltene]

...

To give some concrete examples of the impracticality and ill-advisability of declaring official builds for a source project with so many packages and targets: If there were such a thing as an official place to get binary builds of OpenJDK 8 or 11 sources, where would the official build for a jre of OpenJDK 8u for Apline linux with musl libc (and not glibc) be? Or the official MSI build of a JRE for windows. Or the official deb package for OpenJDK 11.0.3 jdks? None of those can be found in any "official" OpenJDK-tagged place, and I doubt that we should constrain the set of builds people can use by deeming any of the good builds for those "unofficial". any more than you would want Oracle's choices of what they post to superce

I take your point.

There is a serious risk of fragmentation, where no-one knows quite what they're getting when they use an OpenJDK update. The "upstream" builds provide a baseline.

Please step into my shoes for a moment.

I'm not at all complaining about there being some blessed-by-project-lead builds out there. Quite the contrary: I'm quite happy to see them there.

I want to make sure this doesn't fall into the same "there is this 'official' thing controlled by X" trap that you are actively trying to get away from (and seems to have at led, at least partly, to the problems here). This can't be fixed by having "X is not the one official thing, Y is" arguments. A "We don't have an official thing. This is one subset that I know is good, and I'll let others speak for themselves" is a much better stance, reflects reality, and the one that I think you are taking.

Of course, every downstream packager and vendor would prefer their builds to be advertised with "java -version" in every Docker image. Who wouldn't? But as project lead I cannot push Red Hat builds because I have to be impartial. However, there is a screaming need for some known good binaries that I can recommend, and I have solved that problem in the only way I can think of, by producing unbranded vendor-neutral builds. In my opinion they would also provide an excellent basis for Docker images. But yes, Zulu builds would be fine too.

We agree then. Both would be excellent. And some excellent choice should be made, and quick.

[tianon]

@giltene again, I agree that having an Azul Zulu official image would be awesome -- given the amicable conversation happening here, I don't think it makes sense for https://hub.docker.com/_/openjdk to be that, but I do think it makes sense to help that become a thing. Is that something you're interested in pushing forward within https://github.com/docker-library/official-images? Is that something I can help you push foward?

@theRealAph really appreciate your time and attention on this (and your patience with me trying to navigate -- hopefully you can understand some amount of trouble navigating the OpenJDK community in general given the complexity involved)

Definitely happy to take your word for it, especially given https://wiki.openjdk.java.net/display/jdk8u/Main and/or https://wiki.openjdk.java.net/display/JDKUpdates/JDK11u (for the sake of the peanut gallery / those unfamiliar with the project to see your credentials) -- if there's a more authoritative place for us to see who's "involved" and/or someone trustworthy, I think it's useful to have somewhere to point users to say "we're doing this because X said we should and here's where you can see why X is credible" (hence looking for things like the wiki -- if that's not a great place to link and there's somewhere better that displays direct results of voting, etc, any kind of help/pointers in that direction would be awesome). :+1: :heart:

So revising my earlier suggestion, here's what I'd love your thoughts on if you're willing to consider, @theRealAph: :heart:

1. deprecate all openjdk:7 images right away (not mentioned in this thread, but consuming from Debian and so will have similar quality concerns and I don't think there's reasonable builds/support for 7 still that we could consume, correct?)
2. move openjdk:8 and openjdk:11 images to consume from https://adoptopenjdk.net/upstream.html ASAP so that they provide a vanilla upstream experience

Do you think that's a sane path forward? Do you think this is important enough that we should relax our typical backwards compatibility concerns for users and deal with the user fallout separately? (So that we can rectify this in an expedited manner?)

One edge case I see right away is that https://adoptopenjdk.net/upstream.html appears to be JDK-only and we currently provide JRE images for openjdk:8 and openjdk:11 -- do you have any recommendations there? For example, is there any official guideline for what we should strip from those JDK tarballs to produce a stock "JRE" build?

With 12+, we opted to simply drop the JRE builds since there are no artifacts for them published and it is our understanding that jlink is the future of "JRE" use in newer releases, but I don't think we can quite as reasonably simply drop the existing JRE builds without a strong justification (which is why I'm asking). :sweat_smile:

[giltene]

@giltene again, I agree that having an Azul Zulu official image would be awesome -- given the amicable conversation happening here, I don't think it makes sense for https://hub.docker.com/_/openjdk to be that, but I do think it makes sense to help that become a thing. Is that something you're interested in pushing forward within https://github.com/docker-library/official-images? Is that something I can help you push foward?

We'll be happy to make the zulu-openjdk stuff that we've had on docker hub for years "official". But that's an orthoginal thing.

@theRealAph really appreciate your time and attention on this (and your patience with me trying to navigate -- hopefully you can understand some amount of trouble navigating the OpenJDK community in general given the complexity involved)

Definitely happy to take your word for it, especially given https://wiki.openjdk.java.net/display/jdk8u/Main and/or https://wiki.openjdk.java.net/display/JDKUpdates/JDK11u (for the sake of the peanut gallery / those unfamiliar with the project to see your credentials) -- if there's a more authoritative place for us to see who's "involved" and/or someone trustworthy, I think it's useful to have somewhere to point users to say "we're doing this because X said we should and here's where you can see why X is credible" (hence looking for things like the wiki -- if that's not a great place to link and there's somewhere better that displays direct results of voting, etc, any kind of help/pointers in that direction would be awesome). 👍 ❤️

So revising my earlier suggestion, here's what I'd love your thoughts on if you're willing to consider, @theRealAph: ❤️

1. deprecate all openjdk:7 images right away (not mentioned in this thread, but consuming from Debian and so will have similar quality concerns and I don't think there's reasonable builds/support for 7 still that we could consume, correct?)

Umm... OpenJDK 7 is still being updated (and @theRealAph is the project lead for that as well). And there are actively updated builds of 7 by both Azul and Red Hat. It's up to you to decide when to deprecate your 7 images (and I'd agree that we haven't seen much pull for 7 on docker), but the "no reasonable builds for 7 that we could consume" is not a good reason.

2. move openjdk:8 and openjdk:11 images to consume from https://adoptopenjdk.net/upstream.html ASAP so that they provide a vanilla upstream experience

Do you think that's a sane path forward? Do you think this is important enough that we should relax our typical backwards compatibility concerns for users and deal with the user fallout separately? (So that we can rectify this in an expedited manner?)

One edge case I see right away is that https://adoptopenjdk.net/upstream.html appears to be JDK-only and we currently provide JRE images for openjdk:8 and openjdk:11 -- do you have any recommendations there? For example, is there any official guideline for what we should strip from those JDK tarballs to produce a stock "JRE" build?

That's exactly what you are not supposed to be doing. Tested jdk images are not supposed to be torn apart after the fact, hoping that the result is good. You should be using tested jre packages.

And that's the point I was trying to make: You need good bits for your various official docker openjdk images, and at least some of the bits you need are not posted at https://adoptopenjdk.net/upstream.html . Meanwhile Zulu has all the free, well tested openjdk build binary artifacts you need in order to e.g. update your current openjdk:8, openjdk:8-jre, openjdk:8-jre-slim, and openjdk:8-slim to good, actual 8u212 builds.

@theRealAph : Do you consider the Zulu JREs credible, good builds that are fit for that purpose, or have somewhere else he'd recommend you go for a good set of jre packages?

[giltene]

@giltene again, I agree that having an Azul Zulu official image would be awesome -- given the amicable conversation happening here, I don't think it makes sense for https://hub.docker.com/_/openjdk to be that, but I do think it makes sense to help that become a thing. Is that something you're interested in pushing forward within https://github.com/docker-library/official-images? Is that something I can help you push foward?

We'll be happy to make the zulu-openjdk stuff that we've had on docker hub for years "official". But that's an orthoginal thing.

@theRealAph really appreciate your time and attention on this (and your patience with me trying to navigate -- hopefully you can understand some amount of trouble navigating the OpenJDK community in general given the complexity involved) Definitely happy to take your word for it, especially given https://wiki.openjdk.java.net/display/jdk8u/Main and/or https://wiki.openjdk.java.net/display/JDKUpdates/JDK11u (for the sake of the peanut gallery / those unfamiliar with the project to see your credentials) -- if there's a more authoritative place for us to see who's "involved" and/or someone trustworthy, I think it's useful to have somewhere to point users to say "we're doing this because X said we should and here's where you can see why X is credible" (hence looking for things like the wiki -- if that's not a great place to link and there's somewhere better that displays direct results of voting, etc, any kind of help/pointers in that direction would be awesome). 👍 ❤️ So revising my earlier suggestion, here's what I'd love your thoughts on if you're willing to consider, @theRealAph: ❤️

1. deprecate all openjdk:7 images right away (not mentioned in this thread, but consuming from Debian and so will have similar quality concerns and I don't think there's reasonable builds/support for 7 still that we could consume, correct?)

Umm... OpenJDK 7 is still being updated (and @theRealAph is the project lead for that as well). And there are actively updated builds of 7 by both Azul and Red Hat. It's up to you to decide when to deprecate your 7 images (and I'd agree that we haven't seen much pull for 7 on docker), but the "no reasonable builds for 7 that we could consume" is not a good reason.

1. move openjdk:8 and openjdk:11 images to consume from https://adoptopenjdk.net/upstream.html ASAP so that they provide a vanilla upstream experience

Do you think that's a sane path forward? Do you think this is important enough that we should relax our typical backwards compatibility concerns for users and deal with the user fallout separately? (So that we can rectify this in an expedited manner?) One edge case I see right away is that https://adoptopenjdk.net/upstream.html appears to be JDK-only and we currently provide JRE images for openjdk:8 and openjdk:11 -- do you have any recommendations there? For example, is there any official guideline for what we should strip from those JDK tarballs to produce a stock "JRE" build?

That's exactly what you are not supposed to be doing. Tested jdk images are not supposed to be torn apart after the fact, hoping that the result is good. You should be using tested jre packages.

And that's the point I was trying to make: You need good bits for your various official docker openjdk images, and at least some of the bits you need are not posted at https://adoptopenjdk.net/upstream.html . Meanwhile Zulu has all the free, well tested openjdk build binary artifacts you need in order to e.g. update your current openjdk:8, openjdk:8-jre, openjdk:8-jre-slim, and openjdk:8-slim to good, actual 8u212 builds.

@theRealAph : Do you consider the Zulu JREs credible, good builds that are fit for that purpose, or have somewhere else you would recommend they go for a good set of jre packages?

[gnu-andrew]

1. deprecate all openjdk:7 images right away (not mentioned in this thread, but consuming from Debian and so will have similar quality concerns and I don't think there's reasonable builds/support for 7 still that we could consume, correct?)

Umm... OpenJDK 7 is still being updated (and @theRealAph is the project lead for that as well). And there are actively updated builds of 7 by both Azul and Red Hat. It's up to you to decide when to deprecate your 7 images (and I'd agree that we haven't seen much pull for 7 on docker), but the "no reasonable builds for 7 that we could consume" is not a good reason.

But there are no unbranded vanilla builds, as have been recently introduced for 8u & 11u. To be honest, the community interest in 8u & 11u has already far outstripped anything we ever saw for 6 & 7, as I'm sure you're aware from the maintenance work Azul took on with 6. I suspect this is in large part due to the new rapid release cycle & the gulf between 8u & the versions Oracle are rapidly releasing being much wider than the one between 6 and 7, or 7 and 8, making transition harder. The result of this is that work on 6 & 7 has largely been limited to applying the latest security updates to the source tree every quarter.

[gnu-andrew]

On the subject of https://jdk.java.net/, they are, as noted on the page, intended for developers to use as a reference when trying to pass the TCK rather than as something for end use. I had assumed that they were GA versions, but, going on the links for 7u, 8u & 11u (the actual page says little about the version or where the code was sourced from), it's a very odd mix of versions which don't seem to match either GA or the last version Oracle maintained.

In terms of determining "good" and "bad" builds, an important step in that would be to clearly identify what version is being released, what source code it is based on and how it was built.

Maybe I missed it in this thread, but I don't see a link to the Zulu binaries to check if this information is provided there. Is there one?

[giltene]

On the subject of https://jdk.java.net/, they are, as noted on the page, intended for developers to use as a reference when trying to pass the TCK rather than as something for end use. I had assumed that they were GA versions, but, going on the links for 7u, 8u & 11u (the actual page says little about the version or where the code was sourced from), it's a very odd mix of versions which don't seem to match either GA or the last version Oracle maintained.

In terms of determining "good" and "bad" builds, an important step in that would be to clearly identify what version is being released, what source code it is based on and how it was built.

Maybe I missed it in this thread, but I don't see a link to the Zulu binaries to check if this information is provided there. Is there one?

Every single Zulu build binary includes a readme.txt file with a specific code to use to get the complete source code used to produce that specific build. That is as precise as you can get. In addition, Azul provides an actual certificate associated with each package, certifying the fact that the specific package was indeed tested and is certified as Java SE compliant using the associated version TCKs. See https://www.azul.com/downloads/zulu/ for detailed examples of both.

[voitylov]

Each distribution (be it Adopt, Corretto, Red Hat, Zulu, Sap Machine, BellSoft Liberica) has some technical specifics, and it would not be appropriate to substitute one for "official".

I tend to believe Docker official images would have to just reflect the complicated nature of OpenJDK project, with current binaries for 8 and 11 updates available and maintained by different vendors. Yes, that's fragmentation, but this is the situation the upstream project is in. Without first solving the issue of having an official binary of OpenJDK 8 and 11 current updates, I don't think it's appropriate to raise this with Docker.

What can help in the meanwhile is a document jointly created by different vendors that describes the alternatives for docker images. Maybe "Java is Still Free" could be used as a basis for this?

Most respective OpenJDK distributions already have docker images, and it's a matter of getting these into the docker official list. From BellSoft, we suggest all these need to be represented in Docker official list. It's a process that is not specific to Docker. It's not a good comparison, but brew, sdkman, etc have them all.

Another topic that I wanted to raise is quality of Docker containers. If such a situation became possible with official docker containers, maybe the bar for quality to become listed as one of the official docker image of OpenJDK needs to be raised and better defined?

[theRealAph]

In my opinion [the upstream builds ay would also provide an excellent basis for Docker images. But yes, Zulu builds would be fine too.

We agree then. Both would be excellent. And some excellent choice should be made, and quick.

I think the Zulu builds are vanilla too, right? So the only difference is branding and the environment in which the builds were done.

[theRealAph]

So revising my earlier suggestion, here's what I'd love your thoughts on if you're willing to consider, @theRealAph: heart

1. deprecate all `openjdk:7` images right away (not mentioned in this thread, but consuming from Debian and so will have similar quality concerns and I don't think there's reasonable builds/support for 7 still that we could consume, correct?)

That's up to you. 7 is still alive but only receives security updates. We didn't see much pull for 7 build images. Some people still use it, but compatibility between 7 and 8 seems to be good enough in most cases.

2. move `openjdk:8` and `openjdk:11` images to consume from https://adoptopenjdk.net/upstream.html ASAP so that they provide a vanilla upstream experience

Do you think that's a sane path forward? Do you think this is important enough that we should relax our typical backwards compatibility concerns for users and deal with the user fallout separately? (So that we can rectify this in an expedited manner?)

That's very hard for me to comment on. However, @giltene's comment was "Stable debian builds are often dramatically out of date (e.g. the current stable builds are at least 9 months behind on bug fixes and security updates)..." This suggests to me that we're doing users a disservice by holding back updates for long.

One edge case I see right away is that https://adoptopenjdk.net/upstream.html appears to be JDK-only and we currently provide JRE images for openjdk:8 and openjdk:11 -- do you have any recommendations there? For example, is there any official guideline for what we should strip from those JDK tarballs to produce a stock "JRE" build?

JDK 11 has no official JRE build. The reason for this is that the "jlink" tool allows users to create precisely-targeted stripped JDK images for their applications, and this is IMO better for Docker than a JRE image. However, there is a patch for JDK 11 ready (and approved) at https://bugs.openjdk.java.net/browse/JDK-8221610 which will bring back the JRE image.

With 12+, we opted to simply drop the JRE builds since there are no artifacts for them published and it is our understanding that jlink is the future of "JRE" use in newer releases, but I don't think we can quite as reasonably simply drop the existing JRE builds without a strong justification (which is why I'm asking). sweat_smile

11 has never had an official JRE target: I guess the Debian bundles created one somehow.

[theRealAph]

You need good bits for your various official docker openjdk images, and at least some of the bits you need are not posted at https://adoptopenjdk.net/upstream.html . Meanwhile Zulu has all the free, well tested openjdk build binary artifacts you need in order to e.g. update your current openjdk:8, openjdk:8-jre, openjdk:8-jre-slim, and openjdk:8-slim to good, actual 8u212 builds.

@theRealAph : Do you consider the Zulu JREs credible, good builds that are fit for that purpose,

I think I already said so, but I'm only going to endorse Zulu once per thread. :-)

or have somewhere else he'd recommend you go for a good set of jre packages?

We'll put the JDK8 JRE package up on https://adoptopenjdk.net/upstream.html shortly; that's a pretty easy task. There is no JDK11 JRE so it is not possible to post an upstream version of it, but as soon as there is one (which will I expect at the next CPU, a couple of months from now) we'll put it there too.

[mtbc]

Stable debian builds are often dramatically out of date (e.g. the current stable builds are at least 9 months behind on bug fixes and security updates)...

Huh, that surprises me given https://lists.debian.org/debian-security-announce/2019/msg00054.html -- are they perhaps just backporting security fixes to old version numbers?

[tianon]

@theRealAph awesome, that sounds great; thank you!

I think for 11-jre we're probably best off just not having a JRE then -- breakage should be minimal, and we can include jre aliases for the JDK images to keep users functional.

[redbaron]

and we can include jre aliases for the JDK images to keep users functional.

Ubuntu 18:04 tried to lie ot users and it didn't went well. Please don't name things not what they are.

[tianon]

You'd rather break users? 😉

[redbaron]

Yes. Those, who care pull upstream via @sha256: manifest and removing tag would do nothing to them. For the rest publishing 11-jre with

ONBUILD RUN echo 'WARNING, IMAGE YOU ARE TRYING TO USE WAS PULLED OFF DUE TO .... SUGGESTED REPLACEMENT IS ....'; exit 1

Would be most appropriate and honest thing to do.

That is assuming that 11-jre truly doesn't exist and cannot be obtained let's say from Zulu

-------- Original Message -------- On 21 May 2019, 18:12, Tianon Gravi wrote:

You'd rather break users? 😉

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[patrick-webs]

Publishing a JDK image as 11-jre will be confusing and will break users in the future, as they will accidentally use capabilities of JDK image.

Unless it is planned that 11-jre will never again be a JRE image, even after the above mentioned 11 JRE is made available by @theRealAph - "There is no JDK11 JRE so it is not possible to post an upstream version of it, but as soon as there is one (which will I expect at the next CPU, a couple of months from now) we'll put it there too."

[conn]

This may be beside the point, but if you feel like the Debian project isn't maintaining its OpenJDK packages well enough you could try contributing to their processes. I've occasionally contributed the odd patch to Debian and haven't had any negative experiences.

Information about the OpenJDK team and the packages under the responsibility is here.

Debian's distribution of OpenJDK has a good amount of transparency, so maybe "mystery meat" isn't the right term to use here since all components in the package has an identifiable source.

Anyone can see the trackers for OpenJDK packages: https://tracker.debian.org/pkg/openjdk-11

And anyone can browse the code used to package OpenJDK: https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/

With a couple minutes of digging, you can also see which version of OpenJDK the packaging code is pulling from upstream: https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/rules?h=openjdk-11&id=11.0.4%2b2-1#n1668

Debian, like a lot of other free and open source projects, depends on smart folks that are able to identify issues and fix them. Maybe instead of dumping on their efforts we can try to improve them.

This issue feels more like advertising for Azul at the expense of Debian. A Zulu Docker image under the umbrella of official-images would be a great addition, but using cathedral-style distribution of OpenJDK for the ubiquitous OpenJDK Docker image removes transparency and opportunities for the public to participate in improving the software they depend on.

In lieu of Debian, AdoptOpenJDK would probably be the best since they're a vendor-neutral effort and provide all their particulars (tests, build logs, etc.) to the public for inspection: https://github.com/AdoptOpenJDK https://ci.adoptopenjdk.net/

If others would like to use Zulu or any other vendor distribution then they should the official images for them.

[giltene]

Debian, like a lot of other free and open source projects, depends on smart folks that are able to identify issues and fix them. Maybe instead of dumping on their efforts, we can try to improve them.

This issue is not about debian, it's about the official docker images for openjdk and trying to improve them, since as of this writing they are still causing millions of unsuspecting people to run fake, vulnerable 8u212 and 11.0.3 versions.

Specifically, as an end-user, what about this output would make you suspect anything?:

% docker run -it --rm openjdk:8 java -version openjdk version "1.8.0_212" OpenJDK Runtime Environment (build 1.8.0_212-8u212-b01-1~deb9u1-b01) OpenJDK 64-Bit Server VM (build 25.212-b01, mixed mode)

Docker users are (still) being fed this mystery meat every day. And no, the vast majority of them are not likely to suspect the meat is tainted before they eat it, let alone go follow the obscure set of links you include above to figure out how the thing was built, and then <somehow?> deduce from that the thing being built was missing a bunch of bug fixes and security fixes that actual OpenJDK 8u212 (built from actually released sources) include. I (who actually know where to look if I suspect something) only noticed it 48 days after it started doing this...

This issue feels more like advertising for Azul at the expense of Debian.

How is reporting that the official docker images have now repeated the same mystery meat mistake twice in 4 years, or suggesting that the fix is to stop using the builds that have repeatedly caused this even after multiple discussions, and move to a credible OpenJDK build (any credible, well-tested build) an advertisement for Azul? Or even a dig at Debian?

Debian folks have (for years) argued that they are doing nothing wrong by building randomly chosen top of tree samples without labeling their version output as unreleased openjdk binaries. That's completely up to them. The mistake here is for docker official images to dig those builds up and present them to unsuspecting users as presumably released, official openjdk things.

And yes, I did point out that Zulu reliably provides free and credible builds, including the various artifacts needed to immediately address the problem at hand. Others have pointed to credible builds as well. This is the public participating in improving the software they depend on.

A Zulu Docker image under the umbrella of official-images would be a great addition, but using cathedral-style distribution of OpenJDK for the ubiquitous OpenJDK Docker image removes transparency and opportunities for the public to participate in improving the software they depend on.

What cathedral are we talking about here? You think OpenJDK is developed in a cathedral model?

The OpenJDK bazaar sits in the OpenJDK project, where the public participates actively in improving the software java users depend on. That project produces actual source code releases.

Sampling code from the middle of a bazaar development conversation and posting it as the official docker image for openjdk with a fake version label is anything but transparent. It is not a way to encourage public participation in the improvement of the thing. It's a way to make them run for the hills. Defending the practice only makes them run faster.

[conn]

This issue is not about debian, it's about the official docker images for openjdk and trying to improve them

By cutting out Debian as part of the equation and advocating for your own solution.

let alone go follow the obscure set of links you include above to figure out how the thing was built

If you're going to make an open criticism about packaging practices here, on mailing lists, and on Twitter then at least make yourself familiar with how the package in question is built. Given that you maintain your own packages in an apt repository, it would have been safe to assume that you would know more about how repositories are managed in a wider setting.

https://www.google.com/search?q=debian+package+openjdk-11

The Debian Package Tracker is the third link from the top for me. There's visible links on the top left for browsing the source of the packaging code. This really isn't obscure if you at least try.

How is reporting that the official docker images have now repeated the same mystery meat mistake twice in 4 years, or suggesting that the fix is to stop using the builds that have repeatedly caused this even after multiple discussions, and move to a credible OpenJDK build (any credible, well-tested build) an advertisement for Azul? Or even a dig at Debian?

Despite @theRealAph bringing up being impartial in recommending a build, you had still continued to press for yours over one that is vendor-neutral. Instead addressing the actual problem at its source (by helping Debian fix their package), you're campaigning to ignore it. Cutting out this package from the official Docker image reduces its chances of being fixed.

So yes: Azul would stand to gain from this at Debian's expense.

Debian folks have (for years) argued that they are doing nothing wrong by building randomly chosen top of tree samples without labeling their version output as unreleased openjdk binaries. That's completely up to them. The mistake here is for docker official images to dig those builds up and present them to unsuspecting users as presumably released, official openjdk things.

There's no "digging" involved here. Debian and Ubuntu are widely used projects, either directly or indirectly. If what you say is true, more folks than just the ones who use Docker are at risk. If you truly care about this issue, then you will try fixing that ecosystem.

Your company has a lot of valuable expertise about this particular domain; show it by contributing to the solution instead of punching down.

And yes, I did point out that Zulu reliably provides free and credible builds, including the various artifacts needed to immediately address the problem at hand. Others have pointed to credible builds as well. This is the public participating in improving the software they depend on.

You've brought up Zulu multiple times and have even gone as far as asking individual folks for endorsements. You're doing a little more than "pointing out". Injecting a privately-owned process (your build and test procedures) into a widely-used public image instead of another publicly-owned process (Debian or AdoptOpenJDK) doesn't count as a publicly provided solution.

"Docker's OpenJDK image brought to you by Azul" isn't the public participating.

What cathedral are we talking about here? You think OpenJDK is developed in a cathedral model?

The OpenJDK bazaar sits in the OpenJDK project, where the public participates actively in improving the software java users depend on. That project produces actual source code releases.

Sampling code from the middle of a bazaar development conversation and posting it as the official docker image for openjdk with a fake version label is anything but transparent. It is not a way to encourage public participation in the improvement of the thing. It's a way to make them run for the hills. Defending the practice only makes them run faster.

The cathedral is your build and test processes; i.e. the value that you offer. While you readily offer up the source of your packages, the value that you offer is closed to the public. You don't offer build logs. You don't allow the public to see your tests or improve upon them. The most transparency you can offer is build certifications. The public's interaction with your product is a one-way interaction with a download link.

Debian and AdoptOpenJDK's processes are bazaars. You can see their build logs. You can see their tests. You can contribute back to those processes. While both thrive off of contributions from companies such as yours, they are owned by the community.

Currently, all of the software in the OpenJDK is provided by those types of processes. If Zulu were to be included, then it would be the exception. I'm not defending Debian's practice of producing unsafe builds; what I am defending is the culture from which this image and all the software in it is provided.

OpenJDK is important and what Azul does is important to it. Instead of turning your back on a project which everyone can benefit from, please contribute to the solution.

[redbaron]

Instead addressing the actual problem at its source (by helping Debian fix their package)

no, the problem is not with Debian. Problem is with finding and agreeing on a blessed OpenJDK distribution to use. Debian is not as close to OpenJDK project as other candidates (AdoptOpenJDK and Azul), therefore righfuly should be/already is eliminated from this contest.

Think about it, how would you like if "debian:9" image thousands of people pull was not coming from Debian itself, but from some Debian derivative. This is what happened with OpenJDK and should be rectified.

[conn]

The distinction being Debian has a blessed implementation and OpenJDK does not. Until OpenJDK has a blessed implementation, the JDK is going to be derivative no matter how faithful. You have to use the best approximation, which AdoptOpenJDK admittedly is more faithful at doing than Debian. AdoptOpenJDK operates transparently and is open to contributions, similarly to Debian, so using it instead would yield something close to the best of both worlds.

Ultimately, fixing the criteria for Debian's OpenJDK distribution doesn't belong in this issue. I mainly wanted to defend the pros of using packages from a project such as Debian's (collaboration, transparency, yadda yadda yadda) while also reminding folks that the issue upstream isn't out of their control. Debian can definitely use contributions from folks that are opinionated enough to point out very real issues.

There are good reasons to use any Debian package, but in this case AdoptOpenJDK fills the same space as Debian while being more faithfully representative of OpenJDK. If we have to use any distribution to stand in for a blessed implementation, it should probably be that.

[marschall]

You can see their build logs. You can see their tests.

Where? I can go to https://ci.adoptopenjdk.net but how do I find out the build log and test results of a build I download? Test Categorization and Pipelines at AdoptOpenJDK mentions they have closed source private performance tests and post builds even if test fail. Issues like https://github.com/AdoptOpenJDK/openjdk-build/issues/202 also suggest that builds were posted even when tests fail.

When I look at these failing tests:

• https://ci.adoptopenjdk.net/view/Test_openjdk/job/openjdk11_hs_openjdktest_x86-64_windows/218/
• https://ci.adoptopenjdk.net/view/all/job/openjdk11_hs_external_extended_elasticsearch_x86-64_linux/47/

What exactly fails there? Did this also fail with the 11.0.3 release?

[giltene]

This issue is not about debian, it's about the official docker images for openjdk and trying to improve them

By cutting out Debian as part of the equation and advocating for your own solution.

Sort of. By fixing it (urgently) with whatever means are necessary and possible. "Cutting out Debian" (after being contaminated by it twice) is an obvious necessity. These are, literally, the "Docker Official Images for openjdk". Not the "Docker Official Images for debian's builds of the current snapshot of the openjdk source code".

Which solution is used to fix this (again, urgently) is secondary. If you have a place (other than Zulu) to get good, tested bits from today, which will cover the set of official docker openjdk images that are currently exposed, please point to it.

@theRealAph noted that jre builds will be posted to https://adoptopenjdk.net/upstream.html shortly. We can wait for that, or fix things now. Either way, continuing to use the current builds source (debian) will be an obviously bad choice. Your own links to what you found with only "a couple of minutes digging" demonstrate that in detail.

let alone go follow the obscure set of links you include above to figure out how the thing was built

If you're going to make an open criticism about packaging practices here, on mailing lists, and on Twitter then at least make yourself familiar with how the package in question is built. Given that you maintain your own packages in an apt repository, it would have been safe to assume that you would know more about how repositories are managed in a wider setting.

https://www.google.com/search?q=debian+package+openjdk-11

Maybe I don't get the point you are trying to make. Actually following the obscure link above (which no docker end user would be looking for, finding, or following) simply supports the "Mystery Meat" tag label in the original postings. It documents it. Careful reading (which is not what a docker end user would be doing) shows that although the builds linked there appear to carry OpenJDK version numbers (11.0.3, 11.0.4 in this specific case), they are built from unreleased OpenJDK tags, and not from released versions that bear the same (11.0.3, 11.0.4) version numbers. Here is an excerpt portion (to capture the current state as of this day):

Package openjdk-11-jdk
stretch-backports (java): OpenJDK Development Kit (JDK) 
11.0.3+1-1~bpo9+1: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x
buster (testing) (java): OpenJDK Development Kit (JDK) 
11.0.3+1-1: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x
sid (unstable) (java): OpenJDK Development Kit (JDK) 
11.0.4+2-1: alpha amd64 arm64 armel armhf i386 m68k mips mips64el mipsel ppc64 ppc64el s390x sparc64 x32 
11.0.1+13-3 [debports]: sh4

Are you seriously saying that docker official images for openjdk should keep pulling builds from debian repos, even with what you see documented in the page you linked to?

The Debian Package Tracker is the third link from the top for me. There's visible links on the top left for browsing the source of the packaging code. This really isn't obscure if you at least try.

Right. So in-obscure that nobody noticed for 48 days. You seem very upset with the person who noticed it and told everyone else.

Note that I actually don't have an issue with the links being obscure. End users don't need to look for those. I'm just asking for them to post a "beware of leopard" sign in the version string reported by the resulting builds.

“But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a
locked filing cabinet stuck in a disused lavatory with a sign on the
door saying ‘Beware of the Leopard.”

How is reporting that the official docker images have now repeated the same mystery meat mistake twice in 4 years, or suggesting that the fix is to stop using the builds that have repeatedly caused this even after multiple discussions, and move to a credible OpenJDK build (any credible, well-tested build) an advertisement for Azul? Or even a dig at Debian?

Despite @theRealAph bringing up being impartial in recommending a build, you had still continued to press for yours over one that is vendor-neutral.

No. Scroll back to read. I pointed out that you need jre bits for some of the images, and that jre bits are currently available in Zulu and missing from the good alternative. @theRealAph responded "We'll put the JDK8 JRE package up on https://adoptopenjdk.net/upstream.html shortly; that's a pretty easy task." See? Progress. And progress that happened because I pointed to the gap.

Instead addressing the actual problem at its source (by helping Debian fix their package), you're campaigning to ignore it.

Debian is not the source of openjdk bits. It's just one of many downstream builds, which is a demonstrably bad pick for official openjdk bits.

Cutting out this package from the official Docker image reduces its chances of being fixed.

They've had 4.5 years to change their choice of how and where they post unfinished top-of-tree-builds. Time to stop waiting.

Debian folks have (for years) argued that they are doing nothing wrong by building randomly chosen top of tree samples without labeling their version output as unreleased openjdk binaries. That's completely up to them. The mistake here is for docker official images to dig those builds up and present them to unsuspecting users as presumably released, official openjdk things.

There's no "digging" involved here. Debian and Ubuntu are widely used projects, either directly or indirectly. If what you say is true, more folks than just the ones who use Docker are at risk. If you truly care about this issue, then you will try fixing that ecosystem.

I care about the ecosystem. A lot. Enough to rail against mass food poisoning and advocate for urgent recalls when needed. My approach to fixing the ecosystem is to call attention to the existence of Mystery Meat that is not distinguishable as such by it's labeling and to encourage consumers and people in the supply chain to take care to prevent such things from being unknowingly consumed. I actively campaign (and have for many years) for builders of OpenJDK to act responsibly, test their builds, and label them correctly. I've actively worked (with partial success) to expand the availability of OpenJDK Community TCK to OpenJDK builders, and have advised many, including "competitors" that also want to provide good, responsible OpenJDK builds on how to do it right. There are more viable, good builds of OpenJDK now than a few years ago, and I take partial credit for that.

I don't see a strong line between Zulu and other good builds of OpenJDK. I see a very strong line between good builds and bad, mislabeled ones. And I'll keep highlighting the damage bad builds do to unsuspecting Java users and to OpenJDK reputation when they escape into the wild with no warning labels.

Debian's built-from-top-of-tree-and-labeled-the-same-as-releases are (unfortunately) not the only Mystery Meat builds floating around the internet. They are just an existence proof that has been around for 4.5 years and refused to go away. This issue (with the current state of docker official images) is a scary example of what mass contamination looks like when origins and provenance are not properly checked by the people in the supply chain who should be doing so, or if they base their decisions on the false hope that Mystery Meat doesn't exist, or that when it does, it might label itself as such.

Sampling code from the middle of a bazaar development conversation and posting it as the official docker image for openjdk with a fake version label is anything but transparent. It is not a way to encourage public participation in the improvement of the thing. It's a way to make them run for the hills. Defending the practice only makes them run faster.

The cathedral is your build and test processes; i.e. the value that you offer. While you readily offer up the source of your packages, the value that you offer is closed to the public. You don't offer build logs. You don't allow the public to see your tests or improve upon them. The most transparency you can offer is build certifications. The public's interaction with your product is a one-way interaction with a download link.

Debian and AdoptOpenJDK's processes are bazaars. You can see their build logs. You can see their tests. You can contribute back to those processes. While both thrive off of contributions from companies such as yours, they are owned by the community.

Currently, all of the software in the OpenJDK is provided by those types of processes. If Zulu were to be included, then it would be the exception.

You really think that? Where are the build logs and test results for the following (easy to find samples of other, non-debian-sourced official docker openjdk images)? :

https://github.com/docker-library/openjdk/blob/07af3ffded3216b44d69b4f14309fc5a2967e623/12/jdk/oracle/Dockerfile

https://github.com/docker-library/openjdk/blob/d617d7e4b951735c512f3d5ac7672f7394115c91/8/jdk/windows/windowsservercore-ltsc2016/Dockerfile

Or for that matter, where are those non-cathedral-y build and test logs you speak of for the nice well tested builds at https://adoptopenjdk.net/upstream.html ? (which I fully agree, are very good candidates for replacing the debian builds).

[At least two of these are fully TCK tested AFAIK, but that must be have been done in secret cathedrals I guess.]

OpenJDK is important and what Azul does is important to it. Instead of turning your back on a project which everyone can benefit from, please contribute to the solution.

This is my contribution. I have better things I can do with my time, and I do quite a lot of other things in the context of OpenJDK and Java in general. But saving millions of people from being exposed to bad java runtimes somehow matters to me more than to most.

I'll take flak for being the bearer of bad news, and happily let others take credit for playing the "good cop" roles if the combined end result moves things forward. The "good cop" approach alone and the "widely used, must be good" attitudes have clearly not kept this damage from happening. Some amount of public shame and consequences seems to be a good motivating tool that the good cops now hopefully get to leverage.

[conn]

If you feel like publicly shaming all-volunteer organizations is the best way to help them improve, in lieu of actually volunteering, then I'm sure they will gratefully receive such criticism.

[giltene]

If you feel like publicly shaming all-volunteer organizations is the best way to help them improve, in lieu of actually volunteering, then I'm sure they will gratefully receive such criticism.

Docker is not an all-volunteer organization.

[tianon]

I'm not a Docker employee. 😉

[giltene]

I'm not a Docker employee. 😉

Neither am I.

[jozefizso]

It's terrifying to see what we are running in the docker. Nobody has time to track down if somebody in Debian compiles their own copy of source code, which is not official.

And it's more terrifying to see that this is not being addressed in the docker images, which are presented as official and good quality images to use for Java applications.

[tianon]

It is being addressed; the fix is being reviewed (especially since there are a lot of users that it's likely going to break which we're trying to minimize and communicate as best we can). Hoping to have that fix merged and done today.