Java 11.0.20 breaks OpenHAB 3.4.4 (Improved ZIP64 Extra Field Validation)

[RolandRosenfeld]

After upgrade of zulu11 Java from 11.0.19 to 11.0.20 starting OpenHAB 3.4.4 failed starting with messages like

2023-07-21 14:55:10.438 [ERROR] [Events.Framework                    ] - FrameworkEvent ERROR
java.util.zip.ZipException: Exception in opening zip file: /var/lib/openhab/cache/org.eclipse.osgi/18/0/bundleFile
[...]
Caused by: java.util.zip.ZipException: Invalid CEN header (invalid zip64 extra data field size)

This seems to be triggered by the following change in Java 11.0.20 (same with 17.0.8): https://www.oracle.com/java/technologies/javase/11all-relnotes.html

core-libs/java.util.jar ➜ Improved ZIP64 Extra Field Validation (JDK-8302483 (not public)) java.util.zip.ZipFile has been updated to provide additional validation of ZIP64 extra fields when opening a ZIP file. This validation may be disabled by setting the system property jdk.util.zip.disableZip64ExtraFieldValidation to true.

This topic is already discussed in the community forum: https://community.openhab.org/t/openhab-not-working-with-zulu11-0-20/147910

Expected Behavior

OpenHAB should work with recent Java version.

Current Behavior

OpenHAB 3.4.4 fails starting with suggested zulu11 in most recent 11.0.20 version.

Possible Solution

As a workaround it helps adding

EXTRA_JAVA_OPTS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true"

to /etc/default/openhab (for a Debian system, this may be different on other systems).

It may be better to either change the code to support additional ZIP64 extra fields or at least do some Java voodoo to automatically set the above option.

Steps to Reproduce (for Bugs)

1. Start with a working system with OpenHAB 3.4.4 and ZULU11 11.0.19
2. Upgrade ZULU11 to 11.0.20
3. OpenHAB does not work any more.

Your Environment

• Version used: OpenHAB 3.4.4-2 (Debian Package), zulu11 11.0.20-1 (Debian Package), both installed via apt repos.
• Debian 11 (bullseye) on amd64 architecture

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/openhab-not-working-with-zulu11-0-20/147910/9

[rkoshak]

This probably needs to be moved to the openhab-distro repo. I think that's where the EXTRA_JAVA_OPTS are defined/applied.

[RolandRosenfeld]

It seems, that this change in Java was triggered by CVE-2023-22036, so the above option only works around the problem with OpenHAB not starting with the fixed Java version but also disables the security fix in Java. So the correct solution is to implement the ZIP64 checks in OpenHAB instead of disabling the newly introduced check mechanisms in Java.

[rkoshak]

As far as I can tell though there is no code changes involved so the distro repo is still the best place. But perhaps someone knows more than me. I'm still looking at the CVE and the patch info and it doesn't say what end users need to change to accommodate this patch, at least I've not found it yet.

I'm also concerned that we cannot implement this. It's not clear if this means rules will need to be signed in order to run. I don't know if that's feasible.

[Flole998]

It's not clear if this means rules will need to be signed in order to run. I don't know if that's feasible.

No it doesn't, this is only about some header in a zip file being "incorrect". In this case apparently /var/lib/openhab/cache/org.eclipse.osgi/18/0/bundleFile contains a header that is considered bad. I don't know why though and I am also not sure if this affects openHAB 4.0 aswell.

EDIT: Looks like openHAB 4.0 is not affected for some reason, but I am also not sure why older versions are affected.

[mstormi]

So the correct solution is to implement the ZIP64 checks in OpenHAB instead of disabling the newly introduced check mechanisms in Java.

Nope. The JRE you install has to ensure it can run the Java code, you must not ever expect any Java dev to change his code. Proper setup of Java and its parameters is a user's job. If Zulu doesn't work with existing OH code that worked in a previous version and that other JREs work with, that's a problem of Zulu or the Zulu version so adapt your Java parameters on your box or deploy a different JRE - Zulu isn't recommended anyway.

That's not an openHAB (Java app) problem so openhab-core is definitely the wrong place to address this to. I would also not see this as something that openhab-distro is to take care of. If distro sets EXTRA_JAVA_OPTS, those are supposed to be compatible and suitable with any JRE, workarounds for specific OS-JRE combos IMHO don't belong there either. @BClark09 fyi

[rkoshak]

In this case apparently /var/lib/openhab/cache/org.eclipse.osgi/18/0/bundleFile contains a header that is considered bad.

That seems to point to upstream being the problem and not OH itself.

It might already be fixed. I'd recommend clearing the cache and see if that fixes it. If not, an issue on the upstream probably needs to be filed.

EDIT: Looks like openHAB 4.0 is not affected for some reason, but I am also not sure why older versions are affected.

OH 4 is on a newer version of Karaf so I suspect it's not a problem there as newer versions of the library are being used.

If Zulu doesn't work with existing code that worked in a previous version and that other JREs work with, that's a problem of Zulu or the Zulu version so adapt your Java parameters or deploy a different JRE Zulu isn't recommended anyway.

In this particular case, I suspect that Zulu is simply the first to implement a patch for this particular CVE (it's not that highly rated in severity actually). I'd expect it to roll out to the other JRE implementations over time and eventually become a problem for us no matter what Java implementation is installed.

But, it also looks like it's not our problem in the first place but something from upstream. Though until upstream is fixed...

[Flole998]

If code or bundle files don't follow the standard (or even worse are in violation of it) the behavior is undefined, so in that case a Java Developer should (or must) change their code so it adheres to the standard. In this case the zip header is deemed invalid. If it is, that's a bug on the side of whoever created this file. If it isn't, and only if it isn't, this is a bug in Zulu Java.

So the next step would be to check this particular file and examine the header to see why the change causes this exception and if the header or the parser is broken.

[J-N-K]

What is bundle 18 on the system from @RolandRosenfeld? You can list all bundles with list -s -t 0.

[mstormi]

If code or bundle files don't follow the standard

... then we would have seen this on many more systems but we did not. It's likely not even the Java code just a broken package.

First thing @RolandRosenfeld you should do is to a) clear cache and b) reinstall his binary distro to see if the problem persist.

[Flole998]

If code or bundle files don't follow the standard

... then we would have seen this on many more systems but we did not. It's likely not even the Java code just a broken package.

No, since the update is just rolling out since a few days. If there's a single broken package, maybe even an addon, it will take a few days until the reports start coming in. With the release of OH 4.0 they might even never come in since people upgrade before they get hit by the issue.

First thing @RolandRosenfeld you should do is to a) clear cache and b) reinstall his binary distro to see if the problem persist.

Please don't, analyze the issue first and figure out which bundle it is and maybe even grab a copy of the file for further analysis. As the workaround causes it to work now there's no need to rush.

[rkoshak]

I still think:

1. figuring out what bundle is the problem one by running the command @J-N-K posted above to figure out exactly what library is involved followed by
2. clearing the cache to see if the bundle in question has been updated and whether we even have a problem to solve

are the best next steps.

Doing 2 before 1 isn't that big of a deal as if it's solves we'll just never know which bundle temporarily had a bad header. If it doesn't fix it, we've lost nothing.

[RolandRosenfeld]

@J-N-K: You mean in karaf? There I read

 18 │ Active   │  15 │ 4.3.7                   │ org.apache.karaf.features.core  

@mstormi: It was my first try to run openhab-cli clean-cache, but this resulted in the message

2023-07-21 14:34:11.421 [WARN ] [org.apache.felix.fileinstall        ] - Error while starting bundle: file:/usr/share/openhab/addons/org.openhab.binding.worxlandroid-3.4.1.jar
org.osgi.framework.BundleException: Could not resolve module: org.openhab.binding.worxlandroid [20]
  Unresolved requirement: Import-Package: org.openhab.core.common

(this beta binding worked correct before)

Since this looked even more broken to me, I downgraded Java to 11.0.19 and restored last night's backup.

As you can see in https://community.openhab.org/t/openhab-not-working-with-zulu11-0-20/147910 there is at least one other user who run into the same issue.

The bundle file was installed on 2023-05-24 13:31 when I upgraded openhab 3.4.3 to 3.4.4. If someone wants to analyze the file, I just uploaded it to https://www.spinnaker.de/tmp/bundleFile

(I'm not that familiar with Java and toolchain).

[Flole998]

I just upgraded to openHAB 4.0 RC 1 and saw it during the first startup for /var/lib/openhab/cache/org.eclipse.osgi/65/0/.cp/lib/geronimo-osgi-locator.jar. After restarting openHAB again the error disappeared though.

[wborn]

I was able to reproduce it with Eclipse Temurin 17.0.8+7. It does not occur on the first startup but it does happen the next time you start openHAB. :neutral_face:

The issue seems to be with the geronimo-osgi-locator.jar which is a library embedded into the org.apache.aries.spec/org.apache.aries.javax.jax.rs-api/1.0.1 bundle. I also gave version 1.0.4 of the bundle a try but it has the same issue.

On my Ubuntu system I could also easily check that a JAR has issues with:

$ zip -T geronimo-osgi-locator.jar 
META-INF/MANIFEST.MF   bad extra-field entry:
      EF block length (61373 bytes) exceeds remaining EF data (4 bytes)
test of geronimo-osgi-locator.jar FAILED

zip error: Zip file invalid, could not spawn unzip, or wrong unzip (original files unmodified)

It's probably better to disable this new feature for now with https://github.com/openhab/openhab-distro/pull/1559.

[Flole998]

Weird, for me it happened once and since then never again. However, that zip file is broken on my system aswell.

[wborn]

I've created ARIES-2109 to get the root cause fixed.

[wborn]

It would be nice to get rid of these workarounds because apparently the "Improved ZIP64 Extra Field Validation" was a rush job to mitigate CVE-2023-22036. That also explains why JDK-8302483 is not public.

See: https://connortumbleson.com/2023/07/31/invalid-cen-header/

[wborn]

Perhaps some of these workarounds are no longer necessary because the "Improved ZIP64 Extra Field Validation" has been further improved with JDK-8313765 in OpenJDK 11.0.20.1 and 17.0.8.1.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/org-apache-felix-cm-cannot-load-configuration/152872/22