jdk-11.0.7+10 MacOS tar.gz cannot be self bundled in a notarized pkg

[andrew-m-leonard]

Platform: macOS

When trying to create an app bundle that bundles the jdk-11.0.7+10-jre tar.gz bundle, it is not possible to notarize as the the libjli.dylib does not verify

So it seems the issue is that the Contents/MacOS/libjli.dylib fails to codesign --verify:

UNKNOWN:jdk-11.0.7+10-jre andrewleonard$ codesign --verify Contents/MacOS/libjli.dylib 
Contents/MacOS/libjli.dylib: code has no resources but signature indicates they must be present

However.. if I simply move the libjli.dylib to a different directory, it verifies fine! so it seems to be the location of this library that’s the issue. I then discovered if I do a codesign -dv on the jre, it shows this libjli.dylib as the “Executeable”:

UNKNOWN:uu andrewleonard$ codesign -dv jdk-11.0.7+10-jre
Executable=/Users/andrewleonard/Downloads/uu/jdk-11.0.7+10-jre/Contents/MacOS/libjli.dylib
Identifier=libjli
Format=bundle with Mach-O thin (x86_64)
CodeDirectory v=20500 size=786 flags=0x10000(runtime) hashes=16+5 location=embedded
Signature size=9037
Timestamp=28 Apr 2020 at 09:36:30
Info.plist=not bound
TeamIdentifier=VDX7B37674
Runtime Version=10.14.0
Sealed Resources=none
Internal requirements count=1 size=168

Could this be related? causing the codesign --verify to fail on the libjli.dylib ??

[gdams]

running xattr -cr . should fix this:

➜  tmp xattr -cr jdk-11.0.7+10 
➜  tmp codesign --verbose=4 --deep --force -s "Developer ID Application: London Jamocha Community CIC" jdk-11.0.7+10
jdk-11.0.7+10: replacing existing signature
jdk-11.0.7+10: signed bundle with Mach-O thin (x86_64) [net.java.openjdk.11.0.7.jdk]

[hoytk]

Hi, I'm the one that found this. Are you suggesting that I do this to the signed package? Wont that break your signing of the file? Wont this also effect the notarization of the bundle?

[andrew-m-leonard]

@hoytk Hi Kevin, so from talking with @gdams the implication is you need to re-sign the libraries to allow it to be bundled into a new package. As it stands if I understand it correctly, the ..tar.gz contains all AdoptOpenJDK signed libaries and the whole thing is signed as a Bundle with libjli.dylib as the executable. It seems if someone wants to bundle this into a new package they basically have to remove all that and re-sign it, which begs the question why we need to bother signing the tar.gz? I can understand for the .pkg/dmg, but the tar.gz doesn't seem to add any benefit, given bundlers have to remove it. @gdams i've probably got this wrong, but it would be worth setting out the "use cases" for people bundling please? thanks

[andrew-m-leonard]

@hoytk on the "notarization" aspect, the package that you are creating yourself, will need notarizing as a whole at the end anyway.

[hoytk]

What you're shipping is a macOS Application. If you're not going to sign it, you should be shipping just the directory; there is no value in shipping as an Application. It may be this way so it works for the .dmg/.pkg installers. I can use it as is, once the signing is fixed up.

I do NOT need to resign your files. If done correctly, the signing process detects they are already signed and leaves the signature alone. "Sat, 25 Apr 2020 00:12:18 GMT Task: Iterate Directory Contents (libjsig.dylib.dSYM)", "Sat, 25 Apr 2020 00:12:18 GMT Task: Iterate Directory Contents (Contents)", "Sat, 25 Apr 2020 00:12:18 GMT Task: Iterate Directory Contents (Resources)", "Sat, 25 Apr 2020 00:12:18 GMT Task: Iterate Directory Contents (DWARF)", "Sat, 25 Apr 2020 00:12:18 GMT Task: Sign File (libjsig.dylib)", "Sat, 25 Apr 2020 00:12:18 GMT Task: Sign File (libjvm.dylib)", "Sat, 25 Apr 2020 00:12:18 GMT File already signed.",

I'd don't want take on the responsibility of 'proving' the JDK hasn't been modified by signing it Shipping a signed package shows the AdoptOpenJDK has not been tampered with since you published it.

[andrew-m-leonard]

@gdams do you know what set the bundle/app attribute/signing? is it this line? https://github.com/ibmruntimes/openj9-openjdk-jdk11/blob/163e88f2826d2ce6ba74e3fc47174521d0c40cd3/make/MacBundles.gmk#L99

[andrew-m-leonard]

@hoytk @gdams so the issue is caused by the existance of the:

 jdk-11.0.7+10-jre/Contents/Info.plist

Which defines the bundle attributes, if I rename this to Info.plist.old, then libjli.dylib now verifies:

UNKNOWN:Contents andrewleonard$ codesign --verify MacOS/libjli.dylib 
UNKNOWN:Contents andrewleonard$

The jdk-11.0.7+10-jre is then no longer a Bundle obviously, so the following will fail:

UNKNOWN:uu andrewleonard$ codesign -dvvv jdk-11.0.7+10-jre
jdk-11.0.7+10-jre: bundle format unrecognized, invalid, or unsuitable

However, I don't think realistically we can do this, as Mac relies on this "bridging" library to know this jdk/jre provides a JVM for Mac Java Applications.

Doing a bit of googling, I found some references that seem interesting, this article talks about some of the issues: http://openjdk.5641.n7.nabble.com/jlink-and-jpackage-not-producing-correct-macOS-packages-td401931.html#a401972 in particular the comment:

_The code signature thinks that it’s part of a bundle, and yet it has no Info.plist entries or sealed resources.

I’m not sure how it got into this state. On the one hand, it seems that /Volumes/Beat Link Trigger/Beat Link Trigger.app/Contents/runtime was structured to resemble a bundle. On the other hand, it’s doesn’t have an extension (which is typically a key factor in indicating that something is a bundle) and it’s missing its runtime/Contents/_CodeSignature/CodeResources file, which is what contains info about all the resources sealed in the bundle._

If you do a codesign -dvvv MacOS/libjli.dylib it reports:

Info.plist=not bound
Sealed Resources=none

which agrees with what this article is talking about, and is probably why codesign --verify reports error on this library of:

MacOS/libjli.dylib: code has no resources but signature indicates they must be present

So not sure how to proceeed... but to me the Info.plist and Bundle setting/sealed resources do not look correct?

[andrew-m-leonard]

this discussion also indicates the same issue with libjli.dylib: https://github.com/arduino/Arduino/issues/9713#issuecomment-581598678

[andrew-m-leonard]

also doing a basic codesign verify on the jre reports the same error:

UNKNOWN:uu andrewleonard$ codesign -v jdk-11.0.7+10-jre
jdk-11.0.7+10-jre: code has no resources but signature indicates they must be present

[andrew-m-leonard]

another useful quote:

If an app has been signed you can look for a Contents/CodeResources file or a Contents/_CodeSignature/CodeResources file in the bundle. This file lists all the signed components and their expected hash values in the bundle.

[andrew-m-leonard]

to me it's increasingly looking like if we want the *.tar.gz to be a valid Mac Bundle that is signed by AdoptOpenJDK, then we need the above CodeSignature/Resources. Otherwise, it looks like the existing codesign'ing is not of any use, and someone bundling the jre must strip it off and re-sign with their own signature? Which then puts "trust" on the downloaded jre not having being "tampered" with..?

[andrew-m-leonard]

@gdams I am suspecting we need to codesign the whole Bundle at the very end of the build (it can't be modified afterwards), eg:

codesign -s "<AdoptOpenJDK signing identity>" jdk-11.0.7+10-jre

as per instructions here: https://developer.apple.com/library/archive/documentation/Security/Conceptual/CodeSigningGuide/Procedures/Procedures.html In particular the section "What to Code Sign" and bullet: Resources

I will try a test build to see if this makes a difference...

[gdams]

@andrew-m-leonard if you run that command you will lock down the binary and generate a _CodeSignature/CodeResources file. As a result, it then means that users will not be able to change the bundle structure or add Cacerts/jmods etc.

I know it's inconvenient for app developers to have to re-sign the bundle but it's better that way because they are the ones with codesign certs that can make the bundle run again.

If we went for the approach that you've mentioned above, all developers would need a codesign cert to be able to re-certify the binary which is not practical.

[andrew-m-leonard]

@gdams Interesting, yes I see what you mean, as adding custom CA certs into the Java keystore is something typically done, so if we were to lock down the bundle then they would not be able to do that... This implies there's actually no point in signing the tar.gz other than for it being packaged in the .pkg/.dmg. So I think we need to either remove the signing from the .tar.gz artifacts we publish, or document how to remove the signing to correctly..

@hoytk Kevin, so it looks as though you do need to re-sign the jre for this reason, this can be achieved as George mentioned earlier:

xattr -cr jdk-11.0.7+10-jre
codesign --verbose=4 --deep --force -s "<Your codesign identity>" jdk-11.0.7+10-jre

[gdams]

so it looks as though you do need to re-sign the jre

To be very clear this does not re-sign the binaries, it just re signs the bundle

[andrew-m-leonard]

so it looks as though you do need to re-sign the jre

To be very clear this does not re-sign the binaries, it just re signs the bundle

ah thanks George, that wasn't obvious to me!

[hoytk]

so... Seems to me what I want to ship is just the Home directory. There is no need for the rest of the structure as that is only needed for the 'bundle'. This would mean the libjli.dylib would not be shipped. I had done this in an earlier attempt and it didn't effect my usage of Java.

This work from your viewpoint?

If end users (me) need to sign the binaries, can you just not sign them in the first place?

I do see that codesign has a --force flag so that may do what I need.

However... not signing seems like a cleaner choice for the .gz file.

[hoytk]

Let me walk that back a bit. Signing the files seems to work okay. If signed, they are ignored. So the best approach for the .gz file is to just package the Home directory down.

jdk-11.0.7+10-jre/Home/...

[hoytk]

Why does the .gz file need to be a bundle? If it was just a bunch of signed files, I think we'd be good to go.

[andrew-m-leonard]

@gdams am I right in thinking the reason the tar.gz is tagged as a Bundle, is because it is used as the basis for the .pkg ? it has no purpose for the .tar.gz alone and just causes this issue...

[ttomaszewski]

I am having this issue when trying to bundle the .tar.gz file with my electron app. Fails notarization

Contents/Resources/jre/OpenJDK11U-jre_x64_mac.tar.gz/OpenJDK11U-jre_x64_mac.tar/jdk-11.0.7+10-jre/Contents/MacOS/libjli.dylib",
message: "The signature of the binary is invalid.",
docUrl: null,
architecture: "x86_64"

I ran

xattr -cr jdk-11.0.7+10-jre
codesign --verbose=4 --deep --force -s "<Your codesign identity>" jdk-11.0.7+10-jre

prior to building the installer.

[hoytk]

What will signing/notarizing mean for the supported macOS versions? https://adoptopenjdk.net/supported_platforms.html

Do you expect to be able to support anything less than macOS 10.14?

[cypress-benh]

Hopefully this is related to the topic at hand: We're having a signing problem for the jre (jdk8u252-b09-jre) that seems to be related to unintended file duplication. Specifically, the file libjli.dylib seems to be included under both Home and MacOS, as actual instantiated files, rather than the second being a symlink to the first. Further, the files are non-identical. Was this intended, or just an accidental failure to maintain symlinks in the tar file?

[andrew-m-leonard]

Hopefully this is related to the topic at hand: We're having a signing problem for the jre (jdk8u252-b09-jre) that seems to be related to unintended file duplication. Specifically, the file libjli.dylib seems to be included under both Home and MacOS, as actual instantiated files, rather than the second being a symlink to the first. Further, the files are non-identical. Was this intended, or just an accidental failure to maintain symlinks in the tar file?

I believe using a symlink caused Notarization issues, @gdams can you confirm? The libjli.dylib is used as part of the signed Bundle "executable" resource, which I think has to be under MacOS.

@cypress-benh can you give details of the error you are seeing? and your scenario?

[cypress-benh]

@andrew-m-leonard We're re-signing the packages with codesign --force --verbose --options=runtime, and getting something that looks clean, with output saying that it's replacing existing signature, and signed bundle with Mach-O thin (x86_64) [net.java.openjdk.jre]. Later, we're seeing "The signature of the binary is invalid." for the package on Apple's notarization service. The problem goes away for us if we delete that file and create a symlink in the MacOS directory pointing to the file Contents/Home/lib/jli/libjli.dylib. It seems a little strange, too, that the files would differ from one another, but perhaps that's an artifact of signing by AOJDK? I'm new to this whole notarization thing. so thanks for your patience.

[hoytk]

Is there a decision on how to proceed? Ideally the tar file is NOT a bundle but the files are signed.

I put the Home/* directory into my package and it works perfectly.

[andrew-m-leonard]

@karianna @gdams we need some help here, more people are hitting this issue... Should the tar.gz be marked as a Bundle?

[gdams]

@cypress-benh looks like you need to deep sign the package.

codesign --verbose=4 --deep --force -s "Developer ID Application: <cert>"  adoptopenjdk-11.jdk

[cypress-benh]

@gdams Thank you; I'll try that out and report. Re the structure of the tgz, though, with two nonidentical libjli.dylib files under Home and MacOS ... is that intended, as per the upstream comment?

[hoytk]

Just to point out that using --force replaces the signing. This means they are no longer signed with the OpenJDK certificate.

I really think the ideal solution is for the tar file to NOT be a bundle. Just tar up the Home directory... It's then signed and we can include the files in our installer & notarize without a problem.

Seriously, create the bundle so it's ready for the .pkg install, then tar up Contents/Home.

[hoytk]

What is the outlook for fixing this issue?

[andrew-m-leonard]

@hoytk fyi, https://adoptopenjdk.slack.com/archives/C09NW3L2J/p1589910628209200

[andrew-m-leonard]

@hoytk Hi Kevin, I think we need you advice please to proceed this? From experimentation it seems you cannot remove the MacOS directory as doing so invalidates the folder from codesign:

$ codesign -v jdk-11.0.7+10
jdk-11.0.7+10: code object is not signed at all

Removing the Info.plist as well:

$ codesign -v jdk-11.0.7+10
jdk-11.0.7+10: bundle format unrecognized, invalid, or unsuitable

So I am not sure what you were indicating that removing the bundle property would allow codesign to verify? as it would seem codesign can only verify Application Bundles/Apps ..?

Have you managed to proceed in any way with bundling the tar.gz with your package?

thanks Andrew

[andrew-m-leonard]

If I read your comment earlier correctly:

I put the Home/* directory into my package and it works perfectly.

This maybe the answer, as I think it will only codesign -v verify if you copy the jdk "contents" into your own Bundle/App, and then codesign -v your Bundle/App... It won't be possible to verify the jdk.tar.gz alone as it's not been "sealed".

I think we have 2 basic scenarios for consumption of the jdk/jre.tar.gz: 1) Consumer wants to "Configure" their own JDK/JRE with additional/updated Java policies/config, so essentially update the tar.gz content and then re-sign and "seal" as their own "Notarized JDK/JRE" Bundle. ==> They add/update the relavent JDK/JRE files and then deep sign and notarize the Bundle

codesign --verbose=4 --deep --force -s "Developer ID Application: <your_cert>"  <jdk/jre-folder>
Notarize bundle via Apple notarize service

• Result: A sealed Notarized "bundle" configured they way they want.

2) Consumer wants to "embed" the Adopt JDK/JRE in their own Bundle/Application and Notarize ==> Copy the Adopt JDK/JRE bundle content to their own Bundle/Application

cp <jdk/jre>/Contents/Home/* <your_bundle_or_app>
codesign --verbose=4 --deep --force -s "Developer ID Application: <your_cert>"  <your_bundle_or_app>
Notarize bundle via Apple notarize service

• Result: Your own Bundle/App that is Notarized and containing the embedded JDK/JRE

[tresf]

This implies there's actually no point in signing the tar.gz other than for it being packaged in the .pkg/.dmg. So I think we need to either remove the signing from the .tar.gz artifacts we publish, or document how to remove the signing to correctly..

Interestingly, Bellsoft does this exact thing... they provide an unsigned binary download, per https://bell-sw.com/announcements/2020/06/10/How-to-Notarize-a-Mac-Application-with-Liberica-JDK/ quoting:

For the notarization to be successful, it is necessary to get a special build of Liberica JDK 8u252 for MacOS, which allows creating a signed application that will be notarized. In this example, it is called bellsoft-jdk8u252+9-macos-amd64-full-nosign.zip. We use a special unsigned Liberica JDK 8u252 because the javapackager cannot re-sign already signed JDK files.

Note, I'm about knee deep in this process with adopt and I've hit just about every conversation point mentioned. I'm scripting it all using Java.

It may be wise to move over to javapackager to circumvent all this redundant Home relocation logic, but I don't think it's distributed with Adopt yet, right? Worse, it seems to be incompatible with the signed jdk.

Perhaps a better question is... what's the end-goal of this bug report, is it to see the OP through a successful notarization or is there a specific bug with Adopt that needs to be resolved first? I'm asking out of curiosity and also to add myself to the bug notifications. 😄

P.S. I'm using the --force option personally.

[karianna]

I think we were hoping to work with upstream (and Apple) to find a workaround to the issue that Bellsoft neatly describes. That said, I don't think there's anything technically that Adopt can do to resolve this for our binaries. javapackager is something we could investigate/document for folks but I'm unsure if the javapackager with Java 15 can be used to resolve the challenges with 11 and 8

[tresf]

I'm unsure if the javapackager with Java 15 can be used to resolve the challenges with 11 and 8

In my experience, tools like jlink are JDK-specific still, so that might be true. Oddly, Bellsoft's tutorial is for JDK8, so perhaps they backported the tool?

[karianna]

There is meant to be some backward compatibility enablement, but as javapackager is still in flux, I'm not 100% sure whether it works. We'll definitely try though!

[tresf]

There is meant to be some backward compatibility enablement, but as javapackager is still in flux, I'm not 100% sure whether it works. We'll definitely try though!

Slightly off-topic, but what is the state of javapackager and why is it missing from so many JDKs? Is this a tool that OpenJDK is actively trying to backport? I see Oracle tutorials reference it, but I had only assumed that it was a proprietary tool. Once I stumbled upon the Bellsoft article I started to wonder if this tool is what I should have been using all along, since it fixes the Home layout stuff without a bunch of custom scripts (which us packagers are all writing with our own tools/languages).

[karianna]

IIRC It was originally a proprietary tool as part of Java 8. It was then released in Java 9 as javapackager but folks quickly realised it didn't meet all of the requirements and that tool got deprecated in favour of the new javapackage tool (https://openjdk.java.net/jeps/343) which had its beginnings in Java FX. Native packaging is hard, so it's taking a long time to get it right :-)

[hoytk]

If I read your comment earlier correctly:

I put the Home/* directory into my package and it works perfectly.

This maybe the answer, as I think it will only codesign -v verify if you copy the jdk "contents" into your own Bundle/App, and then codesign -v your Bundle/App... It won't be possible to verify the jdk.tar.gz alone as it's not been "sealed".

I think we have 2 basic scenarios for consumption of the jdk/jre.tar.gz:

1. Consumer wants to "Configure" their own JDK/JRE with additional/updated Java policies/config, so essentially update the tar.gz content and then re-sign and "seal" as their own "Notarized JDK/JRE" Bundle. ==> They add/update the relavent JDK/JRE files and then deep sign and notarize the Bundle

codesign --verbose=4 --deep --force -s "Developer ID Application: <your_cert>"  <jdk/jre-folder>
Notarize bundle via Apple notarize service

• Result: A sealed Notarized "bundle" configured they way they want.

1. Consumer wants to "embed" the Adopt JDK/JRE in their own Bundle/Application and Notarize ==> Copy the Adopt JDK/JRE bundle content to their own Bundle/Application

cp <jdk/jre>/Contents/Home/* <your_bundle_or_app>
codesign --verbose=4 --deep --force -s "Developer ID Application: <your_cert>"  <your_bundle_or_app>
Notarize bundle via Apple notarize service

• Result: Your own Bundle/App that is Notarized and containing the embedded JDK/JRE

Sorry I got lost in other stuff :-)

I think if people want a BUNDLE, they should 'pkgutil --expand-full' the pkg installer and do what they want. They'll sign & notarize it as theirs.

The tar file should be targeted for consumers (like me) so it can be included in our installers as just a bunch of files. You sign the binaries and we include them in our install... and we're done.

[aahlenst]

It's unlcear what the expectations are and, as it stands, there's no action to take. If anyone comes up with an actionable request, I'm happy to reopen.

[tresf]

It's unlcear what the expectations are and, as it stands, there's no action to take. If anyone comes up with an actionable request, I'm happy to reopen.

I think mimicking other JDK providers and providing an unsigned binary is a viable action. Another actionable item is documenting the --force and complex notarization steps. To that point, Oracle's Java would suffer the same fate, so this may all be better handled as a medium, stackoverflow or blog as it's more of a challenge of bundling signed binaries at large, not necessarily JDK binaries, so I tend to agree with the closure.

[aahlenst]

@tresf Is https://blog.adoptopenjdk.net/2020/05/a-simple-guide-to-notarizing-your-java-application/ sufficient?

[tresf]

@aahlenst absolutely. :)