MacOS hardened runtime support

[tkie]

I would like to bundle the AdoptOpenJDK into a MacOS application. To use Apple's notary service all executables in my app have to be codesigned with hardened runtime support.

The ApoptOpenJDK executables are codesigned already, but unfortunately without hardened runtime support.

I can re-sign them myself like this:

codesign --verbose --options runtime --force --sign "Developer ID Application: SecSign Technologies Inc." bin/java

but apparently this damages the executables. For example "java" is not able to print its version any more:

java --version
Error occurred during initialization of VM
Could not reserve enough space in CodeHeap 'non-nmethods' (2496K)

Would the ApoptOpenJDK team be able to enable the hardened runtime ("--options runtime") during the codesigning in the build process of the AdoptOpenJDK? This would be great.

Thank you in advance for any comments on this issue.

Kind regards Tilo

[gravelld]

Is that the entitlements for the runtime or for the app that bundles the runtime?

[dg76]

I have used it for all codesign commands, e.g. for the dmg but also for all other files (dylib, jar, .app):

codesign --timestamp --entitlements src/main/deploy/package/macosx/MyApp.entitlements --options runtime --deep -vvv -f --sign "Developer ID Application: John Public (XXXXXXXXXX)" MyApp-1.0.dmg

[gravelld]

@dg76 thanks - I'll try that. I haven't written any entitlements before now, but since Feb 3rd everything seems to have broken; maybe I was relying on default that used to be acceptable, but now aren't (obviously that applies to the hardened runtime and builds against old SDKs; which is why I am trying to adopt this build).

[gravelld]

That appears to work, thanks! So, to document my end solution:

% security unlock-keychain -p passwordhere codesigning.keychain
% find my-app.app -type f \
  -not -path "*/Contents/runtime/*" \
  -not -path "*/Contents/MacOS/my-app" \
  -not -path "*libapplauncher.dylib" \
  -exec codesign --timestamp --entitlements /tmp/bliss.entitlements -s "XXX" --prefix com.myapp. --options runtime -v --keychain /path/to/codesigning.keychain {} \;

% find my-app.app/Contents/runtime -type f \
  -not -path "*/legal/*" \
  -not -path "*/man/*" \
  -exec codesign -f --timestamp --entitlements /tmp/bliss.entitlements -s "XXX" --prefix com.myapp. --options runtime -v --keychain /path/to/codesigning.keychain {} \;

% codesign -f --timestamp --entitlements /tmp/bliss.entitlements -s "XXX" --prefix com.myapp. --options runtime -v --keychain /path/to/codesigning.keychain my-app.app/Contents/runtime

% codesign -f --timestamp --entitlements /tmp/bliss.entitlements -s "XXX" --prefix com.myapp. --options runtime -v --keychain /path/to/codesigning.keychain my-app.app

All tests work:

% codesign -vvv --deep --strict my-app.app/Contents/runtime 
my-app.app/Contents/runtime: valid on disk
my-app.app/Contents/runtime: satisfies its Designated Requirement
% codesign -vvv --deep --strict my-app.app/                
--prepared:/private/tmp/my-app.app/Contents/MacOS/libapplauncher.dylib
--validated:/private/tmp/my-app.app/Contents/MacOS/libapplauncher.dylib
my-app.app/: valid on disk
my-app.app/: satisfies its Designated Requirement
% spctl -a -t exec -vv my-app.app          
my-app.app: accepted
source=Developer ID
origin=XXX

Does it run?

% my-app.app/Contents/MacOS/my-app 

Yes!

Does it notarize?

Status: success
Status Code: 0
Status Message: Package Approved

Hurrah!

Summary for those bundling a JRE:

• jpackage signing is broken right now, if you want to get something notarized
• You must sign your own code.
• You must re-sign the JRE. Use -f to force the signature.
• You must specify entitlements in all signing calls

[ijabz]

Hi, can I just ask do we currently have a Mac build for Java 8 or Java 9 that allows it to be notarized according to the new rules or not ?

[addsomebass]

@ijabz if I'm understanding your question correctly, you want to bundle a jvm within an application, and then notarize that. Currently, you can't. The latest AdoptOpenJDK build is built with an older version of the Mac OS SDK and can't be repackaged and re-notarized.

If you just want to install java, the latest .pkg installers will work on OSX Catalina, and I can install and execute Java just fine. These installers have been notarized, and can install Java to your machine without issue.

gadams is working on getting a notarizable build of Java 8, and is hoping to have that done at the end of the month. I'm wishing him and everyone else luck.

[ijabz]

if I'm understanding your question correctly, you want to bundle a jvm within an application, and then notarize that. Currently, you can't

Yes correct, okay this will require code changes on my part but does the AdoptOpenJDK Java 11 build work ?

Or if I install Oracle builds will that work, I am currently using Oracle Java 8 on MacOS and that did work until Feb 3rd, is this what you mean by a .pkg build ?

I'm just looking for a solution that I can use now, Im very unclear on what the options currently are.

[addsomebass]

@ijabz Sorry I can't speak to Java 11 or later, as I have not worked with it personally. Judging by the comments here it seems like there is a way to get a working bundle. I can only speak to working with Java 8 builds.

The mac installer packages, or .pkg files, are installers for software. Currently there are some .pkg installers available for AdoptOpenJDK builds that are notarized and will install Java correctly, like this one here: https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u242-b08/OpenJDK8U-jdk_x64_mac_hotspot_8u242b08.pkg

[ijabz]

I feel Im being dumb but I dont understand that on one side AdoptOpenJDk are working on a Java 8 build that is notarized but that wont be ready until end of the month, but conversely there is already an AdoptOpenJDk Java 8 installer that will already install a notarised version Java 8 ?

[dg76]

@ijabz I think the reason is that if something has been notarized before February 3rd 2020, it is still valid and still works on macOS Catalina. However it wouldn't be possible to notarize it again now, because the rules are more strict now. So the AdoptOpenJDK Java 8 installer was probably just notarized before February 3rd and thus can still be used.

But you cannot package it any more because since February 3rd Apple requires apps to be "hardened", which requires XCode 10. And Java 8 has not been compiled with XCode 10, so it cannot be hardened (yet).

So you can install AdoptOpenJDK Java 8 despite it is not hardened just because it was notarized before February 3rd. But now it wouldn't be possible any more to create a new version of it or to package a program with it, because it cannot be hardened and thus cannot be notarized.

From what I have read at least JDK 11.0.7 and JDK 14 seem to work fine, are compiled using XCode 10, are "hardened" and at least JDK 14 can be used to package programs.

[ijabz]

Okay thankyou I understand better now. One more thing when you refer to JDK 11.0.7 and JDK 14 I assume you mean AdoptOpenJDk builds, is the situation for AdoptOpenJDk builds the same as for the downloads available on the Oracle site ?

[dg76]

Sorry, I don't know. I have tried it only with OpenJDK (not AdoptOpenJDK but a different distribution but I guess it is the same for all OpenJDK distributions). However because it seems to require changes in JDK 8 to make it compatible with XCode 10 and, as far as I know, Oracle makes the same changes in OpenJDK and their own distribution, I would think that their JDK 8 isn't hardened either (or otherwise they would have released an OpenJDK version of Java 8 that would be hardened, too). But as I said I haven't tried it.

[ijabz]

ok, thanks. I ask because Im not entirely sure what the advantage of using AdoptOpenJdk build is rather than just downloading from Oracle.

[aahlenst]

@ijabz Oracle JDK is only free to use for limited set of use cases (see Oracle Java SE Licensing FAQ). OpenJDK builds by Oracle are available, too, and still free to use without restrictions, but the number of supported platforms and versions is significantly lower than what we offer here at AdoptOpenJDK. As the name implies, AdoptOpenJDK are builds of OpenJDK. One of our main objectives is to upstream all the changes. Other vendors of OpenJDK (and there are many: Amazon, Azul, BellSoft, SAP, …) may have different policies.

[ijabz]

Right, and I see they only seems to offer the MacOS build for the latest builds now, okay thanks.

[pierucci18]

@gdams, do you still expect to have a notarized jdk8 binary ready by the end of the month? Also, would that include the J9 JVM?

[peslys]

@gdams, do you still expect to have a notarized jdk8 binary ready by the end of the month? Also, would that include the J9 JVM?

Can you try BellSoft JDK8 distribution (https://bell-sw.com/pages/java-8u242/). I replaced AdoptOpenJDK J8 with BellSoft J8, get rid off "The binary uses an SDK older than the 10.9 SDK." and finaly got notarized.

Looks like BellSoft J8 is already notarized ready.

[iRobertas]

As I understand, BellSoft JDK is not free..

[peslys]

As I understand, BellSoft JDK is not free..

The GNU General Public License (GPL) with "CLASSPATH" EXCEPTION TO THE GPL for Regular Liberica Java SE 8u242 JDK or JRE the same license as OpenJDK.

They offer commercial support if you need.

[iRobertas]

Thanks for clarification !

[derek-baxter]

That appears to work, thanks! So, to document my end solution:

% security unlock-keychain -p passwordhere codesigning.keychain
% find my-app.app -type f \
  -not -path "*/Contents/runtime/*" \
  -not -path "*/Contents/MacOS/my-app" \
  -not -path "*libapplauncher.dylib" \
  -exec codesign --timestamp --entitlements /tmp/bliss.entitlements -s "XXX" --prefix com.myapp. --options runtime -v --keychain /path/to/codesigning.keychain {} \;

% find my-app.app/Contents/runtime -type f \
  -not -path "*/legal/*" \
  -not -path "*/man/*" \
  -exec codesign -f --timestamp --entitlements /tmp/bliss.entitlements -s "XXX" --prefix com.myapp. --options runtime -v --keychain /path/to/codesigning.keychain {} \;

% codesign -f --timestamp --entitlements /tmp/bliss.entitlements -s "XXX" --prefix com.myapp. --options runtime -v --keychain /path/to/codesigning.keychain my-app.app/Contents/runtime

% codesign -f --timestamp --entitlements /tmp/bliss.entitlements -s "XXX" --prefix com.myapp. --options runtime -v --keychain /path/to/codesigning.keychain my-app.app

All tests work:

% codesign -vvv --deep --strict my-app.app/Contents/runtime 
my-app.app/Contents/runtime: valid on disk
my-app.app/Contents/runtime: satisfies its Designated Requirement
% codesign -vvv --deep --strict my-app.app/                
--prepared:/private/tmp/my-app.app/Contents/MacOS/libapplauncher.dylib
--validated:/private/tmp/my-app.app/Contents/MacOS/libapplauncher.dylib
my-app.app/: valid on disk
my-app.app/: satisfies its Designated Requirement
% spctl -a -t exec -vv my-app.app          
my-app.app: accepted
source=Developer ID
origin=XXX

Does it run?

% my-app.app/Contents/MacOS/my-app 

Yes!

Does it notarize?

Status: success
Status Code: 0
Status Message: Package Approved

Hurrah!

Summary for those bundling a JRE:

• jpackage signing is broken right now, if you want to get something notarized
• You must sign your own code.
• You must re-sign the JRE. Use -f to force the signature.
• You must specify entitlements in all signing calls

I was able to follow your steps however I have a library and even with <key>com.apple.security.cs.disable-library-validation</key> <true/>

I get this error:

Exception in thread "JavaFX Application Thread" java.lang.UnsatisfiedLinkError: /private/var/folders/0f/trlp6tp95qjbzm99ds8pt8zm0000gp/T/JxBrowser/7.5/libbrowsercore_toolkit.dylib: dlopen(/private/var/folders/0f/trlp6tp95qjbzm99ds8pt8zm0000gp/T/JxBrowser/7.5/libbrowsercore_toolkit.dylib, 1): no suitable image found. Did find: /private/var/folders/0f/trlp6tp95qjbzm99ds8pt8zm0000gp/T/JxBrowser/7.5/libbrowsercore_toolkit.dylib: code signature in (/private/var/folders/0f/trlp6tp95qjbzm99ds8pt8zm0000gp/T/JxBrowser/7.5/libbrowsercore_toolkit.dylib) not valid for use in process using Library Validation: mapping process and mapped file (non-platform) have different Team IDs

Does this mean the library I have also needs that entitlement for it to work or is it something on my end?

[adamulrich]

The most frustrating part of the notarization process is that I've had to plumb this release process into my CI/CD pipeline. I wish apple would supply a test endpoint. I don't need it notarized, I just need to know if it is notarizable. Granted the difference is small, but a couple things make it uncomfortable.

1. If I consume an update to a 3rd party, I need an intake process that validates that the update is notarizable. By submitting that 3rd party for validation, I've now notarized that 3rd party using my companies creds, and my security folks think that isn't awesome.
2. I'm submitting things for notarization every day that I absolutely do not intend to provide to a customer, and I have to use credentials that that are authorized to do more than just test. Because those creds are now part of my pipeline, my security surface area for a breach in credentials is larger. So I want a test endpoint that uses no creds, or uses creds that are for testing purposes only.

[derek-baxter]

Is anyone else running into the issue I am having @gravelld @dg76 I have tried creating an entitlement but that doesn't seem to disable library validation

[gravelld]

@dcboy95 I'm sorry - I don't know how particular entitlements work with respect to coverage over the entire bundle. I found, with the code signing, I had to resign the entire library I shipped with, so maybe that means the entire codebase you ship with also has to have the entitlement? Sorry.

[dg76]

@dcboy95 I would think that you just have to resign the file "libbrowsercore_toolkit.dylib" using codesign and your own certificate. (I thought I had already posted this suggestion here but I cannot find any response from me.) The error just seems to indicate that the library has a different signature than the rest of the application. And I think everything has to have the same signature to ensure that no one else put anything into it. So just try to run codesign on "libbrowsercore_toolkit.dylib". Does that fix it?

[derek-baxter]

@dg76 Unfortunately that didn't work. With the third party library it reinstalls itself if there is a different version. So when I codesign that file, it sees it as a old file and downloads a new one, replacing the modified one. Resulting in the same error. I thought the entitlement com.apple.security.cs.disable-library-validation would fix this issue however it doesn't seem to.

[dg76]

@dcboy95 I think this won't work. I think the purpose of Apple's notarization system is to ensure that the developer has confirmed that all parts of his app have been tested and are fine. Loading parts from other developers during runtime, which have not been tested and confirmed by the developer, are probably not allowed. Can't you just disable the automatic download function?

[derek-baxter]

@dg76 Thanks for the information, I will have to research their documentation to see if this is a possibility. Thanks again

[aahlenst]

AdoptOpenJDK 14+36 is out which is fully notarized. Please report back in case of troubles.

[aahlenst]

A fully notarized JDK 8 is still in the works. We hope to get this included in the next quarterly CPU (closest Tuesday to the 17th of April).

The patches required passed some test runs (including the TCK), but more validation is necessary. @gdams plans to publish the current state so that we can provide builds for testing in the not so distant future.

[derek-baxter]

@aahlenst is there any plans for JDK 11?

[aahlenst]

@dcboy95 A fully notarized JDK 11 will be published as part of the next quarterly CPU (shortly after the closest Tuesday to the 17th of April if I remember Oracle's rule correctly). In the meantime, you can grab a nightly build of JDK 11 on https://adoptopenjdk.net/nightly.html?variant=openjdk11&jvmVariant=hotspot and report back if it doesn't work.

[NishikaDeSilva]

Hi I tried using jdk-11.0.7+7 for the installer but still [1] fails with th error "The signature of the binary is invalid"

[1] /Contents/MacOS/libjli.dylib

[gdams]

@NishikaDeSilva try running this:

codesign --verbose=4 --deep --force -s "Developer ID Application: MyTeam (XXXX)" installer.jdk

[NishikaDeSilva]

@gdams I tried your solution.

command: codesign --verbose=4 --deep --force -s "Developer ID Application: abc (xxxxx)" jdk-11.0.7+7

output: jdk-11.0.7+7: replacing existing signature jdk-11.0.7+7: resource fork, Finder information, or similar detritus not allowed

But it indicates the following error... jdk-11.0.7+7: code has no resources but signature indicates they must be present.

Am I doing something wrong ?

[MoxxiManagarm]

I tried 11.0.7+7 from the nightly build. Notarization passes and the application comes up successfully. However, it seems that binaries (dylib) are now looked up in the executed jar bundle only by any native load that is used by JNA. As a result our application now receives UnsatisfiedLinkErrors when trying to access dylib installed on the system. This was working before.

[aahlenst]

@MoxxiManagarm Are the dylibs signed? Check with codesign -v -v /path/to/dylib.

[MoxxiManagarm]

@aahlenst I doubt, those are 3rd party dylibs (device drivers) installed on the system, they are not part of our application. I also have the entitlement active com.apple.security.cs.disable-library-validation, that shouldn't be an issue.

The lib exists within the directory /usr/local/lib/

[justin-espedal]

Using the nightly build of JDK 11's JRE, I have the same issue as NishikaDeSilva. I've tested nightly builds of JRE 13 and 14 as well for good measure.

I don't know if any of this will be helpful, but here are my notes.

Running codesign -dvvv libjli.dylib to check the signature results in this.

Executable=[...]/Contents/MacOS/libjli.dylib
Identifier=libjli
Format=bundle with Mach-O thin (x86_64)
CodeDirectory v=20500 size=786 flags=0x10000(runtime) hashes=16+5 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha1=512c6006363d2928665d9d135f360f974fe66d27
CandidateCDHashFull sha1=512c6006363d2928665d9d135f360f974fe66d27
CandidateCDHash sha256=1157d57e9849eb161251e3a4bca6e2ab7200eaa1
CandidateCDHashFull sha256=1157d57e9849eb161251e3a4bca6e2ab7200eaa11cf4d1f13ae558f0a8ae207e
Hash choices=sha1,sha256
CMSDigest=2727b2d669fee540fd5848fffba07d583d737065d4f28c205530b29f44dfb347
CMSDigestType=2
CDHash=1157d57e9849eb161251e3a4bca6e2ab7200eaa1
Signature size=9037
Authority=Developer ID Application: London Jamocha Community CIC (VDX7B37674)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Mar 31, 2020 at 8:44:15 AM
Info.plist=not bound
TeamIdentifier=VDX7B37674
Runtime Version=10.14.0
Sealed Resources=none
Internal requirements count=1 size=168

Verifying the signature with spctl -a -vv libjli.dylib results in this: libjli.dylib: code has no resources but signature indicates they must be present

I can't find any good documentation on that error, but it seems to have something to do with "sealed resources", which the signature above says is none. Doesn't make much sense.

Attempting to codesign the file myself results in this: resource fork, Finder information, or similar detritus not allowed

All existing documentation on that error says that it's caused by extended attributes on the file. Specifically, com.apple.FinderInfo and com.apple.ResourceFork. But ls -l@ libjli.dylib and xattr -l libjli.dylib lists no xattributes.

Curiously, moving the file to another location (anywhere outside the Contents/MacOS folder) or changing the file extension to something other than .dylib will make codesigning and signature verification pass normally, without either of the above nonsensical errors. The codesign mechanism must be taking the location of the file into account as well, causing it to differ from the results of the file at Contents/Home/lib/jli/libjli.dylib.

[MoxxiManagarm]

Any update on the driver dylib which is not found within /usr/local/lib/ after notarization (nightly build)?

[gdams]

Verifying the signature with spctl -a -vv libjli.dylib results in this: libjli.dylib: code has no resources but signature indicates they must be present

@justin-espedal try running this on the top-level directory:

xattr -cr .
codesign --verbose=4 --deep --force -s "Developer ID Application: MyTeam (XXXX)" adoptopenjdk-11.jdk 

[gdams]

output: jdk-11.0.7+7: replacing existing signature jdk-11.0.7+7: resource fork, Finder information, or similar detritus not allowed

@NishikaDeSilva run this on the directory before running the codesign command:

xattr -cr .

[justin-espedal]

That does work. No need for the recursive -cr, just removing com.apple.FinderInfo from the top-level folder is enough. It makes sense that the FinderInfo attribute on the top-level folder is what interferes with codesigning of the entire package.

Regardless of whether that xattr is present or not, the libjli.dylib still can't be codesigned on its own. Does the codesign mechanism requires dylib's in that specific location to verify something with the containing packages code signature? If that's the case (or perhaps regardless of that), why isn't the whole package simply codesigned by default?

I'd certainly prefer not to codesign the entire JRE's code if I don't have to.

[Ram-Krishnamurthy]

@gadams @aahlenst Any update on the notarized JDK 1.8? Are we releasing it on Apr 17th? Is there an earlier kit to try out?

[aahlenst]

Next week, there will be a 8u252 without notarization and hardened runtime first. Afterwards, we are going to flip the switches and build one with notarization and hardened runtime. Additional information will follow. I do not know whether there is already a binary that is ready for testing by others.

[ijabz]

Will 8u252 include libAppleScriptEngine.dylib that is built against later sdk so doesnt fail with 'The binary uses an SDK older than the 10.9 SDK', this would be most helpful,https://stackoverflow.com/questions/61208189/java-notarization-of-libapplescriptengine-dylib-failing-with-the-binary-uses-an

[meshcow]

The long-awaited 8u252 is out, but it turned out almost all binaries in it were still built with sdk 10.8, which prevents successful notarization. The JRE from https://ci.adoptopenjdk.net/view/work%20in%20progress/job/jdk8u-mac-x64-hotspot-notarized/10/ contained 10.14 binaries so we hoped the official 8u252 also will.

Next week, there will be a 8u252 without notarization and hardened runtime first. Afterwards, we are going to flip the switches and build one with notarization and hardened runtime.

Do you guys have a schedule for this you can share, please?

[aahlenst]

@meshcow Hard to estimate. That's why I gave no estimate in the first place. Pipelines with the regular builds are still running (for example, 14.0.1 with HotSpot and 11.0.7 with OpenJ9). The version with hardened runtime will be built afterwards.

[karianna]

@meshcow - we'll be building an 8u252.1 to test our Mac Os notarization patch, should be here in a day or two.

[ijabz]

FYI I used BellSoft 8u252 to try and get a libAppleScriptEngine.dylib that is built against later sdk to allow notarization, and can confirm that worked.