Notarization fails on MacOS 10.15.3

[Ankitr19]

We are trying to notarize our electron app, but the notarization is not working . It fails with the following issue :-

"issues": [ { "severity": "error", "code": null, "path": "/Contents/Resources/app/node_modules/fsevents/build/Release/.node", "message": "The binary is not signed.", "docUrl": null, "architecture": "x86_64" }, { "severity": "error", "code": null, "path": "/Contents/Resources/app/node_modules/fsevents/build/Release/.node", "message": "The signature does not include a secure timestamp.", "docUrl": null, "architecture": "x86_64" } ]

How to resolve this error, and also if we have a method for deep code signing in electron-builder?

[twigs67]

I'm having the same issue with both fsevents and Puppeteer. @Ankitr19 did you figure this out?

Also, are you using electron-notarize?

[dboakes]

Started having the same issue within the last week. For me it's python binaries within app.asar.unpacked. Presumably Apple have changed how they handle notarization?

[timfish]

Notarisation has started failing for us for the same reason.

{
  "logFormatVersion": 1,
  "jobId": "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxx",
  "status": "Invalid",
  "statusSummary": "Archive contains critical validation errors",
  "statusCode": 4000,
  "archiveFilename": "OurApp.zip",
  "uploadDate": "2020-02-10T12:43:07Z",
  "sha256": "9517c159237a6c18c16d439894eb16abec73757c5e194833bdec7a35bad661a2",
  "ticketContents": null,
  "issues": [
    {
      "severity": "error",
      "code": null,
      "path": "OurApp.zip/OurApp.app/Contents/Resources/app.asar.unpacked/node_modules/fsevents/build/Release/.node",
      "message": "The binary is not signed.",
      "docUrl": null,
      "architecture": "x86_64"
    },
    {
      "severity": "error",
      "code": null,
      "path": "OurApp.zip/OurApp.app/Contents/Resources/app.asar.unpacked/node_modules/fsevents/build/Release/.node",
      "message": "The signature does not include a secure timestamp.",
      "docUrl": null,
      "architecture": "x86_64"
    }
  ]
}

[twigs67]

Ok, I think I narrowed it down. This tweet/video helped me.

So it looks like we have to sign each of the individual packages. Now the question is, how do we do that? Any guesses @timfish @dboakes @Ankitr19 ?

[timfish]

It looks like Apple tightened up notarisation on Feb 3rd and warnings became errors. Because warnings are not displayed by electron-notarize we're only finding out about them now.

For electron-builder macOS configuration there is a binaries option but looking at it's usage, these need to be an array of full paths to each binary. electron-builder is already detecting binaries for relocation to app.asar.unpacked so it should ideally add these to the list of files to automatically sign.

[twigs67]

@timfish I was about to mention electron-builder's option to add additional binaries. I'm not having any luck with it though right now, but I don't think I have the correct paths.

I'm using, myApp.zip/myApp.app/Contents/Resources/node_modules/puppeteer/.local-chromium/mac-706915/chrome-mac/Chromium.app/Contents/MacOS/Chromium

[timfish]

I think the signing step occurs before packaging so the path would be more like ./node_modules/puppeteer/.local-chromium/mac-706915/chrome-mac/Chromium.app/Contents/MacOS/Chromium.

I've also seen some logic in electron-builder that finds *.app within and signs them too. I'm pretty sure you want to be signing Chromium.app rather than the internal binary.

I'm in the process of running some CI builds with DEBUG=electron-builder to get some helpful debug output.

[twigs67]

I'm trying out your suggestions now. I also found this plist param that seems to allow you to skip code signing for "arbitrary" plugins & frameworks. Not entirely sure what that means, but I tried it and it didn't work.

[timfish]

It seems like that might be a runtime option but you never know, maybe the notarisation service pays attention to it?

Edit We already have com.apple.security.cs.disable-library-validation: true so I doubt that will fix it.

[twigs67]

@timfish I'm not having any luck. I've tried a number of different ways. Are you having any luck?

I also noticed that sometimes I get this error and sometimes I don't.

    {
      "severity": "error",
      "code": null,
      "path": "myApp.zip/myApp.app/Contents/Resources/node_modules/fsevents/build/Release/.node",
      "message": "The signature does not include a secure timestamp.",
      "docUrl": null,
      "architecture": "x86_64"
    }

I don't even have fsevents in my package.json. I have it in my yarn.lock file for some reason. I tried deleting it, but it still pops up. So obviously when I try adding it to the list of binaries, it says it can't find it.

[timfish]

Nope, no luck working around this for now.

fsevents is likely a sub-dependency of one of your dependencies. yarn why fsevents should tell you why its included.

[dboakes]

I tried messing around with afterPack yesterday, ran a function to manually codesign the binaries that are coming causing the issue, but like you said, says the path doesn't exist.

I'm all out of ideas as well, but desperate for a fix!

[kzimny]

Can you post your electron-builder config part from package.json? Do you build for MAS or without any target specified?

[dboakes]

"build": {
    "afterSign": "deploy/notarize.js",
    "appId": "com.electron.APP_NAME",
    "buildDependenciesFromSource": true,
    "generateUpdatesFilesForAllChannels": true,
    "icon": "build/assets/icon.png",
    "productName": "APP_NAME",
    "directories": {
      "buildResources": "build",
      "output": "dist"
    },
    "mac": {
      "hardenedRuntime": true,
      "gatekeeperAssess": false,
      "entitlements": "build/entitlements.mac.plist",
      "entitlementsInherit": "build/entitlements.mac.plist",
      "category": "public.app-category.sports",
      "target": "zip"
    },
    "asarUnpack": [
      "build/pydist"
    ],
    "publish": [
      {
        "provider": "github",
        "owner": "OWNER_NAME",
        "repo": "APP_NAME",
        "token": "TOKEN",
        "private": true,
        "releaseType": "release"
      }
    ]
  }

[kzimny]

Try to unpack all .node executable files and put them in a folder called app.asar.unpacked. You do this by adding:

"asarUnpack": [
  "**/*.node"
]

Then the files will be signed and Apple will be able to read the signature.

[timfish]

@kzimny we are using that option and it doesn't appear to help. Our config is:

productName: **snip**
appId: **snip**
copyright: Copyright © **snip**
directories:
  buildResources: '../build'
  output: '../releases'
  app: './'
files:
  - '**/*'
  - '!**/*.{map,dll,so,dylib,cat,inf,sys,exe}'
  - '!**/node_modules/**/*.{cc,c,h,obj,pdb,map,lib,tlog,m4,sh,S}'
  - '!**/node_modules/**/doc/**'
  - '!**/node_modules/**/configure'
afterPack: ../build/afterpack.js
afterSign: ../build/notarize.js
asarUnpack:
  - '**/*.node'
win:
  **snip**
linux:
  **snip**
mac:
  hardenedRuntime: true
  darkModeSupport: true
  extraFiles:
    - from: ../build/${os}/${arch}
      to: ./resources/
      filter:
        - '**/*'
  target:
    - target: dmg
      arch:
        - x64

and we're using the latest electron-builder and electron-notarize.

Since Apple tightened up their notarisation on 3rd of Feb, we've started getting this error. I've run some builds with DEBUG=electron-builder where I can see the final sign step but it doesn't look like it's signing any other binaries.

[kzimny]

Just for a test, try to remove the target and all excluded files:

files:
  - '**/*'

[dboakes]

Made no difference for me sadly. For me, the binaries that are showing as unsigned, are within the python binary I am already unpacking.

[timfish]

Just for a test, try to remove the target and all excluded files

Build is running in CI. Will let you know how it goes. I could dump the debug output here but it would take me a while to sanitise it!

[timfish]

No, notarisation still fails with the same error with the simplified config.

Oh, one point worthy of note is that we don't get any notarisation errors about the other 5 native modules we're using (keytar, usb-detection, level, ref and ffi) which are all direct dependencies of our app. The error only lists fsevents which is a dependency of chokidar on macOS. So maybe the other native modules are getting signed? There is nothing in the debug output to suggest they are getting signed but Apple isn't complaining about them.

I've got to leave my desk for much of the rest of the day but later I'll try a build with chokidar excluded and see if that fixes it.

[twigs67]

but like you said, says the path doesn't exist.

@dboakes I debugged the CI build yesterday and fsevents doesn't become available until some point after the code signing event and I'm not sure how to remove chokidar from the build as @timfish suggested.

I tried manually code signing each binary in travis-ci, but I was unable to add the keychain to "list-keychains".

Has anyone tried to codesign manually?

[twigs67]

I was able to successfully notarize the app, but I've made so many changes I'm not sure what did it. However, I made the following changes prior to:

• Removed build.extraResources: [ 'node_modules'] from package.json
• Added asarUnpack
• Updated osx_image in .travis.yml to xcode11.3 (part of the new requirements state you have to have SDK version 10.9 or above, I was on 10

package.json

"build": {
    "asar": true,
    "mac": {
      "target": [
        "dmg",
        "zip"
      ],
      "hardenedRuntime": true,
      "entitlements": "./build/entitlements.mac.plist",
      "entitlementsInherit": "./build/entitlements.mac.plist",
      "gatekeeperAssess": true,
      "cscLink": "build/DevApp_cert.p12",
      "cscInstallerLink": "build/devInstaller_cert.p12"
    },
    "afterSign": "./build/afterSignHook.js",
    "dmg": {
        "sign": false
    }
  },

entitlements.mac.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
    <true/>
    <key>com.apple.security.cs.allow-dyld-environment-variables</key>
    <true/>
    <key>com.apple.security.cs.disable-library-validation</key>
    <true/>
  </dict>
</plist>

I also just ran a build without asarUnpack (because that causes path issues in my app) and it worked.

Update: my app is now not running correctly, because it cannot find Puppeteer. It's included in the package, but it can't reach it. It only seems to work if I add it to extraResources, but then it gives me the issue we've all been dealing with.

[kzimny]

Great. Thank you for the information.

[Christilut]

Same problem here. I tried asarUnpack but seemed to make no difference in the errors that Apple reports in their log. Also tried with disable-library-validation.

It's mostly tripping over extraResources that I added, all the .so and .dylib files.

[timfish]

We're using extraFiles over extraResources but we're not seeing any errors about anything we copy there.

[twigs67]

@timfish I tried using extraFiles instead of extraResources, but my app still can't locate Puppeteer. Can you think of any way around this?

Is there a way to add these directories to the system path so that it will look there when being imported? I see the files option has to and from, but I'm not sure how I'm supposed to use that.

[timfish]

@twigs67 I've not solved these issues yet for our app yet so I'm not really in a position to advise!

Today I'm going to put together a simple reproduction of this issue. This is not being helped by the fact that the npm registry is returning 404 for 7zip-bin which is a dependency for electron-builder!

[Christilut]

What does work is codesign these extra resources manually, something like this: codesign -s "Company Name" --options=runtime my-file The Apple server no longer complains about them then.

I'm still figuring out if I should write a script that codesigns everything or some other method. But this is a temporary workaround at the very least.

[timfish]

Our signing is all on CI and uses CSC_LINK and CSC_KEY_PASSWORD so it's going to be a pain to get manual codesign working with that.

I'd rather get this fixed in electron-builder so everyone can benefit from it.

[Christilut]

All my library files are signed and work but I can't sign some executable files:

    {
      "severity": "error",
      "code": null,
      "path": "App.zip/App.app/Contents/Resources/lib/foo.zip/foo",
      "message": "The signature of the binary is invalid.",
      "docUrl": null,
      "architecture": "x86_64"
    },

Even though it's signed on my mac. I tried zipping it with ditto -c -k --sequesterRsrc --keepParent but that had no effect.

[Christilut]

Okay the binary needed the entitlement added to pass validation, that wasn't necessary for the other files: codesign -s "Company Name" --entitlements ../../build/entitlements.mac.plist --options=runtime foo

Now all files are signed properly and accepted by Apple.

The above ditto command isn't needed for zip files in extraResources.

[dboakes]

Okay the binary needed the entitlement added to pass validation, that wasn't necessary for the other files: codesign -s "Company Name" --entitlements ../../build/entitlements.mac.plist --options=runtime foo

Now all files are signed properly and accepted by Apple.

The above ditto command isn't needed for zip files in extraResources.

So when you say you're codesigning the file manually, this is before both building and packing? Or after build and before packing?

So would we need to sign the files in the node_modules folder? And if so, presumably we'd have to resign anytime we update the node_module package, or clear and rebuild node_modules etc?

[Christilut]

I did this before the building and before the packing. Basically before running any build script.

I don't know about how node_modules works in this case. I made sure I signed all files that were in the sign logs from Apple. In my case mostly .so and .dylib files. You should check those logs and see which files are the problem.

You can use something like this to sign them: find . -name *.so -exec codesign -s "Company Name" --options=runtime -v {} \;

[dboakes]

That's fixed it for me. Ideally we'll have a built in solution via electron-builder soon, but for now that'll work just fine. I had to tweak it slightly at the end to make it work.

Ended up with: find . -name *.so -exec codesign -s "Company Name" --entitlements path/to/entitlements.mac.plist --options=runtime -v '{}' +

[timfish]

At least the OP @Ankitr19 and I are getting the same error about fsevents and now I've downloaded the partial build output I've noticed something about the path in the error from Apple:

Contents/Resources/app.asar.unpacked/node_modules/fsevents/build/Release/.node.

In the build output, the path to the fsevents binary is actually:

Contents/Resources/app.asar.unpacked/node_modules/fsevents/build/Release/fse.node

There is no binary called .node.

I've checked out our app and electron-builder has automatically signed all the binaries. This fsevents error appears to be nothing to do with electron-builder failing to sign binaries.

[timfish]

Right, I've tracked this down further.

When I look inside /Contents/_CodeSignature/CodeResources I can see the erroneous entry for the non-existent .node binary:

<key>Resources/app.asar.unpacked/node_modules/fsevents/build/Release/.node</key>
<dict>
  <key>hash</key>
  <data>9IzMKqONHghSkplT82jF/PnLpUE=</data>
  <key>hash2</key>
  <data>/NbNi+UkFPdmUkO1DmybZTd6KoE8CSYMDKphFKASr2U=</data>
</dict>
<key>Resources/app.asar.unpacked/node_modules/fsevents/build/Release/fse.node</key>
<dict>
  <key>hash</key>
  <data>zuLJVwlfQYli6Cl3z79bBgQUSG4=</data>
  <key>hash2</key>
  <data>nH7XA4DJplkSZb8kURHFduJMeD+SG8k/O6M+agSjxZs=</data>
</dict>

So electron-builder is including this invalid entry when code signing. @develar any idea where I should look to fix this?

[timfish]

I've checked the debug output with DEBUG=electron-osx-sign and it lists all the binaries it's signing in the app. Everything looks good and there is no erroneous node_modules/fsevents/build/Release/.node entry listed there.

I have no idea how the erroneous entry is ending up in _CodeSignature/CodeResources because that file is written by the codesign tool. 🤷‍♂️

[twigs67]

I'm lost as well, but I'm working on a slightly different problem than you @timfish.

However, maybe this will help you. I noticed that I don't get the fsevents code signing error when I set "asar":false. Maybe the asar packaging is adding it?

[ching2018]

Has anyone solved this problem?

[timfish]

I worked around it by finding a way to remove fsevents as a dependency.

[ching2018]

我通过找到一种将其删除fsevents为依赖项的方法来解决此问题。

I use puppeteer whith Chromium.The fsevents is not in devDependencies.

[twigs67]

Ya, I found a workaround for this as I wasn't able to get the Chromium binary to sign or Apple to take it. @HoldSkill are you able to notarize your app with Chromium?

[Ankitr19]

We resolved the issue, by moving fsevents as a Dev dependency.

[parachvte]

I've checked the debug output with DEBUG=electron-osx-sign and it lists all the binaries it's signing in the app. Everything looks good and there is no erroneous node_modules/fsevents/build/Release/.node entry listed there.

@timfish I did set DEBUG=electron-osx-sign as you mentioned, seems like the electron-osx-sign ignores executable files that are hidden (start with dot), while two other files (fsevents.node and fse.node) are not ignored:

...
electron-osx-sign Signing... /Users/path-to-my-app.app/Contents/Resources/app/node_modules/fsevents/bin/darwin-x64-57/fsevents.node +302ms
electron-osx-sign Signing... /Users/path-to-my-app.app/Contents/Resources/app/node_modules/fsevents/build/Release/fse.node +297ms
...

And I checked the fsevents folder in node_modules/fsevents/build/Release with ls -al, both fse.node and .node executables are listed there.

[parachvte]

After read through the source code of electron-osx-sign, I think the problem is that in https://github.com/electron/electron-osx-sign/blob/master/util.js#L227, it rejectes hidden file when walking through Contents folder.

case '': // Binary
  if (path.basename(filePath)[0] !== '.') {
    return getFilePathIfBinaryAsync(filePath)
  } // Else reject hidden file
  break

@develar @sethlu Have any clue that why hidden files are ignored, and why they are still included in _CodeSignature/CodeResources? There is a related PR with this issue open since 2018.

[timfish]

@parachvte It looks like electron-builder vendors electron-osx-sign here.

[parachvte]

@timfish Yes @develar copied electron-osx-sign here in order to solve MAS build on July 17 2019. Later on the two branches get maintained independently. I don't know how it goes but the source code I posted above remains the same in both repos.

Edit: Finally we resolve this issue by moving fsevents dependencies to devDependencies. Common related packages like:

• chokidar
• babel-cli
• nodemon
• stylint
• webpack (<-- we misplaced some webpack deps in dependencies

[sethlu]

I think there are kind of 2 questions brought up here?

1. Why are hidden files ignored?
2. Why is the PR https://github.com/electron/electron-osx-sign/pull/169 there?

The historical/original intention for hidden files is that files that .gitignore or .DS_Store can be skipped because they aren't really executable binaries. Also, funnily they don't have an extension name because the dot-DS_Store is its name (as far as path.basename() concerns). The actual binary executables also don't usually have an extension. So the decision was made to skip all the hidden-file-looking files.

However, this turned out to be overlook as reported in https://github.com/electron/electron-osx-sign/issues/168 not because native node modules are ignored and causing code-signing issues, but because the compiled native code modules can simply be named as .node and placed in a dist directory (which was unheard of, at that time, to me). And the current version of electron-osx-sign@0.4.15 still skips the .node file, but you can force-code-sign that file by passing an argument to electron-osx-sign.

Therefore, https://github.com/electron/electron-osx-sign/pull/169 is created so that we check all files if they are binary-looking and we code-sign them all. However, I didn't have much time to test it, so it's left as an open PR. Theoretically, it should work. I'll be curious to hear if this PR works for people so it can be merged to master without causing troubles for existing workflows that people set up.

The _CodeSignature/CodeResources are artifacts from code-signing. And you may find more information about them here: https://developer.apple.com/library/archive/documentation/Security/Conceptual/CodeSigningGuide/AboutCS/AboutCS.html

Hope this helps!

[eschirtz]

To add my two cents..

I'm also experiencing similar issues with deep code signing, this time with @tensorflow/tfjs-node.

{
severity: "error",
code: null,
path: "Samply_Staging.zip/Samply (Staging).app/Contents/Resources/app.asar.unpacked/node_modules/@tensorflow/tfjs-node/deps/lib/libtensorflow.so",
message: "The binary is not signed.",
docUrl: null,
architecture: "x86_64"
},
{
severity: "error",
code: null,
path: "Samply_Staging.zip/Samply (Staging).app/Contents/Resources/app.asar.unpacked/node_modules/@tensorflow/tfjs-node/deps/lib/libtensorflow.so",
message: "The signature does not include a secure timestamp.",
docUrl: null,
architecture: "x86_64"
},
{
severity: "error",
code: null,
path: "Samply_Staging.zip/Samply (Staging).app/Contents/Resources/app.asar.unpacked/node_modules/@tensorflow/tfjs-node/deps/lib/libtensorflow_framework.so",
message: "The binary is not signed.",
docUrl: null,
architecture: "x86_64"
},
{
severity: "error",
code: null,
path: "Samply_Staging.zip/Samply (Staging).app/Contents/Resources/app.asar.unpacked/node_modules/@tensorflow/tfjs-node/deps/lib/libtensorflow_framework.so",
message: "The signature does not include a secure timestamp.",
docUrl: null,
architecture: "x86_64"
}

I've read through this thread a couple times now and can't quite find "the fix". Wondering if anyone has any input, or if any help is needed to get this resolved in electron-builder itself.

Thanks!

[eschirtz]

After inspecting _CodeSignature/CodeResources

I'm seeing entries for the two binaries that apple is complaining about, and I can't see why they are any different than any of the other files...

<key>Resources/app.asar.unpacked/node_modules/@tensorflow/tfjs-node/deps/lib/libtensorflow.so</key>
<dict>
    <key>hash</key>
    <data>
    9U5XsHIre9vGzF+oKWbZLjs2yM0=
    </data>
    <key>hash2</key>
    <data>
    Icrwobv2XnX5PewitMAGEA6XMbYB1C3IqQ+DbXQfzZk=
    </data>
</dict>
<key>Resources/app.asar.unpacked/node_modules/@tensorflow/tfjs-node/deps/lib/libtensorflow_framework.so</key>
<dict>
    <key>hash</key>
    <data>
    DvQTWVYAdgjJtJGJPNEyi6Xl9u0=
    </data>
    <key>hash2</key>
    <data>
    6CixRVOadOYI5f37uAXKK4RHAhED63ZVWCU6vgglxJk=
    </data>
</dict>

Again, any advice would be awesome!