search

electron / universal

Create Universal macOS applications from two x64 and arm64 Electron applications
MIT License
112 stars 43 forks source link

Can't reconcile two non-macho files #41

Open WrathofthePast opened 2 years ago

WrathofthePast commented 2 years ago

In macOS 12.3, Apple removed Python 2. electron-builder v23.0.0 added Python 3 support to resolve the issue. However, electron-universal now fails to package the universal app.

Full Error Message:

  • electron-builder  version=23.0.0 os=21.4.0
  • loaded configuration  file=package.json ("build" field)
  • loaded parent configuration  file=/Users/me/razer-macos/node_modules/electron-webpack/out/electron-builder.js
  • writing effective config  file=dist/builder-effective-config.yaml
  • executing node-gyp rebuild  platform=darwin arch=x64
  • packaging       platform=darwin arch=x64 electron=12.2.3 appOutDir=dist/mac-universal--x64
  • executing node-gyp rebuild  platform=darwin arch=arm64
  • packaging       platform=darwin arch=arm64 electron=12.2.3 appOutDir=dist/mac-universal--arm64
  • packaging       platform=darwin arch=universal electron=12.2.3 appOutDir=dist/mac-universal
  ⨯ Can't reconcile two non-macho files build/Makefile  failedTask=build stackTrace=Error: Can't reconcile two non-macho files build/Makefile
    at Object.exports.mergeASARs (/Users/BobWhite/razer-macos/node_modules/@electron/universal/src/asar-utils.ts:151:13)
    at exports.makeUniversalApp (/Users/BobWhite/razer-macos/node_modules/@electron/universal/src/index.ts:200:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at MacPackager.doPack (/Users/BobWhite/razer-macos/node_modules/app-builder-lib/src/macPackager.ts:125:9)
    at MacPackager.pack (/Users/BobWhite/razer-macos/node_modules/app-builder-lib/src/macPackager.ts:179:7)
    at Packager.doBuild (/Users/BobWhite/razer-macos/node_modules/app-builder-lib/src/packager.ts:441:9)
    at Object.executeFinally (/Users/BobWhite/razer-macos/node_modules/builder-util/src/promise.ts:12:14)
    at Packager._build (/Users/BobWhite/razer-macos/node_modules/app-builder-lib/src/packager.ts:376:31)
    at Packager.build (/Users/BobWhite/razer-macos/node_modules/app-builder-lib/src/packager.ts:337:12)
    at Object.executeFinally (/Users/BobWhite/razer-macos/node_modules/builder-util/src/promise.ts:12:14)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
./dist/mac-universal/Razer macOS.app: No such file or directory

Does electron-universal directly utilize Python 2?

indutny-signal commented 2 years ago

Hello!

This is a feature I worked on. It looks like build/Makefile is different when you are building for x64 than it is for arm64. Any idea why?

indutny-signal commented 2 years ago

The easiest fix might be to block it from being put into asar from package.json's by putting "!build/Makefile" into build.files[] array.

WrathofthePast commented 2 years ago

The easiest fix might be to block it from being put into asar from package.json's by putting "!build/Makefile" into build.files[] array.

I tried that and got a different error:

  ⨯ Can't reconcile two non-macho files build/addon.target.mk  failedTask=build stackTrace=Error: Can't reconcile two non-macho files build/addon.target.mk
    at Object.exports.mergeASARs (/Users/BobWhite/razer-macos-backup/node_modules/@electron/universal/src/asar-utils.ts:151:13)
    at exports.makeUniversalApp (/Users/BobWhite/razer-macos-backup/node_modules/@electron/universal/src/index.ts:200:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at MacPackager.doPack (/Users/BobWhite/razer-macos-backup/node_modules/app-builder-lib/src/macPackager.ts:125:9)
    at MacPackager.pack (/Users/BobWhite/razer-macos-backup/node_modules/app-builder-lib/src/macPackager.ts:179:7)
    at Packager.doBuild (/Users/BobWhite/razer-macos-backup/node_modules/app-builder-lib/src/packager.ts:441:9)
    at Object.executeFinally (/Users/BobWhite/razer-macos-backup/node_modules/builder-util/src/promise.ts:12:14)
    at Packager._build (/Users/BobWhite/razer-macos-backup/node_modules/app-builder-lib/src/packager.ts:376:31)
    at Packager.build (/Users/BobWhite/razer-macos-backup/node_modules/app-builder-lib/src/packager.ts:337:12)
    at Object.executeFinally (/Users/BobWhite/razer-macos-backup/node_modules/builder-util/src/promise.ts:12:14)
error Command failed with exit code 1.

Interestingly, this is a similar error to the one I got when I used electron-universal v1.0.5 with electron-builder v23.0.0. However, the first two at ...s are different. In that electron-builder version, they used your mergeASARs API, so electron-universal v1.0.5 didn't satisfy the dependency.

indutny-signal commented 2 years ago

Well, you can ignore that other files as well, or even all files in build except build/Release? Not sure how it is used in your app, but the basic idea here is that you shouldn't bundle platform-specific files into your app unless they:

WrathofthePast commented 2 years ago

Well, you can ignore that other files as well, or even all files in build except build/Release?

That's what I've trying the past few minutes. After adding `"!build/addon.target.mk", the same error occurred with build/config.gypi, so added that too. The next error is with build/Release/addon.node. What do I do now since you're saying I can't ignore files in build/Release?

Not sure how it is used in your app, but the basic idea here is that you shouldn't bundle platform-specific files into your app unless they:

  • have different names on each platform

They do not.

  • are binary .node files (binding for native addons)

Do you mean like node-addon-api or native-ext-loader?

indutny-signal commented 2 years ago

Could you run file build/Release/addon.node? I'm curious if this is an addon built for macOS at all.

WrathofthePast commented 2 years ago

Could you run file build/Release/addon.node? I'm curious if this is an addon built for macOS at all.

What do you mean? Run it in the command line or add it to package.json?

By the way, I added "!build/Release/addon.node". Then it errored on build/Release/.deps/Release/obj.target/addon/src/driver/addon.o.d. After adding that it errored on each driver in build/Release/.deps/Release/obj.target/addon/librazermacos/src/lib. Finally, after adding that it, it successfully packaged the universal app and it works.

indutny-signal commented 2 years ago

What do you mean? Run it in the command line or add it to package.json?

command line.

WrathofthePast commented 2 years ago

What do you mean? Run it in the command line or add it to package.json?

command line.

Here's the output:

macbook-pro:razer-macos-backup me$ file build/Release/addon.node
build/Release/addon.node: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64Mach-O 64-bit bundle x86_64] [arm64:Mach-O 64-bit bundle arm64Mach-O 64-bit bundle arm64]
build/Release/addon.node (for architecture x86_64): Mach-O 64-bit bundle x86_64
build/Release/addon.node (for architecture arm64):  Mach-O 64-bit bundle arm64

I didn't need to exclude build/binding.Makefile, build/gyp-mac-tool, or build/Release/obj.target to make it build. I had to exclude build/Release/.deps, build/Release/addon.node, build/Makefile, build/addon.target.mk, and build/config.gypi.

indutny-signal commented 2 years ago

Ah gosh, it is already universal. I wonder why it was different between x64 and arm64 bundles then...

WrathofthePast commented 2 years ago

Ah gosh, it is already universal. I wonder why it was different between x64 and arm64 bundles then...

I have an idea: in my build.files[] array, I've always included build. Is that redundant? Also, I noticed that build/Release/obj.target is identical to build/Release/.deps/Release/obj.target. Why are they duplicated?

indutny-signal commented 2 years ago

This is just the way the node.js addons are built. You can definitely include whole folder if this makes sense for your app.

WrathofthePast commented 2 years ago

I noticed in addon.target.mk that -arch x86_64 is not being passed to CFLAGS_Release while -arch arm64 is. Furthermore, I noticed config.gypi contains "msvs_configuration_platform": "ARM64" & "xcode_configuration_platform": "arm64", but does not have x86_64 anywhere. Do you think my binding.gyp is misconfigured?

indutny-signal commented 2 years ago

Unclear. I'd be more curious to learn what made this binding universal. Normally node addons are for single arch.

WrathofthePast commented 2 years ago

Unclear. I'd be more curious to learn what made this binding universal. Normally node addons are for single arch.

I attached the package.json and binding.gyp, if you wanted to look at it. Do you think all the exclusions from the build.files[] array will any cause problems? package.json and binding.gyp.zip

WrathofthePast commented 2 years ago

Not sure how it is used in your app, but the basic idea here is that you shouldn't bundle platform-specific files into your app unless they:

  • have different names on each platform
  • are binary .node files (binding for native addons)

Unclear. I'd be more curious to learn what made this binding universal. Normally node addons are for single arch.

I think I figured it out. I use native-ext-loader instead of node-addon-loader. In contrast, it "allows to build path at runtime." In my webpack.main.additions.js, native-ext-loader is set to resolve: {extensions: ['.node']}. Doesn't this mean I have platform-specific binary .node files?

oltreseba commented 2 years ago

I'm having the same problem:

 ⨯ Can't reconcile two non-macho files node_modules/better-sqlite3/build/deps/sqlite3.target.mk  failedTask=build stackTrace=Error: Can't reconcile two non-macho files node_modules/better-sqlite3/build/deps/sqlite3.target.mk
    at Object.exports.mergeASARs (/Users/oltreseba/Akiflow/akido/node_modules/@electron/universal/src/asar-utils.ts:151:13)
    at exports.makeUniversalApp (/Users/oltreseba/Akiflow/akido/node_modules/@electron/universal/src/index.ts:200:13)
    at MacPackager.doPack (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/macPackager.ts:125:9)
    at MacPackager.pack (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/macPackager.ts:179:7)
    at Packager.doBuild (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:441:9)
    at Object.executeFinally (/Users/oltreseba/Akiflow/akido/node_modules/builder-util/src/promise.ts:12:14)
    at Packager._build (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:376:31)
    at Packager.build (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:337:12)
    at Object.executeFinally (/Users/oltreseba/Akiflow/akido/node_modules/builder-util/src/promise.ts:12:14)

Everything was working fine until i upgraded electron builder from 22.14.13 to 23.0.2.

oltreseba commented 2 years ago

After excluding .mk files from the build, i'm not having problem with a .a file: (not sure if i will be fine in excluding this file).

⨯ Can't reconcile two non-macho files node_modules/better-sqlite3/build/Release/sqlite3.a  failedTask=build stackTrace=Error: Can't reconcile two non-macho files node_modules/better-sqlite3/build/Release/sqlite3.a
    at Object.exports.mergeASARs (/Users/oltreseba/Akiflow/akido/node_modules/@electron/universal/src/asar-utils.ts:151:13)
    at exports.makeUniversalApp (/Users/oltreseba/Akiflow/akido/node_modules/@electron/universal/src/index.ts:200:13)
    at MacPackager.doPack (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/macPackager.ts:125:9)
    at MacPackager.pack (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/macPackager.ts:179:7)
    at Packager.doBuild (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:441:9)
    at Object.executeFinally (/Users/oltreseba/Akiflow/akido/node_modules/builder-util/src/promise.ts:12:14)
    at Packager._build (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:376:31)
    at Packager.build (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:337:12)
    at Object.executeFinally (/Users/oltreseba/Akiflow/akido/node_modules/builder-util/src/promise.ts:12:14)
WrathofthePast commented 2 years ago

After excluding .mk files from the build, i'm not having problem with a .a file: (not sure if i will be fine in excluding this file).

⨯ Can't reconcile two non-macho files node_modules/better-sqlite3/build/Release/sqlite3.a  failedTask=build stackTrace=Error: Can't reconcile two non-macho files node_modules/better-sqlite3/build/Release/sqlite3.a
    at Object.exports.mergeASARs (/Users/oltreseba/Akiflow/akido/node_modules/@electron/universal/src/asar-utils.ts:151:13)
    at exports.makeUniversalApp (/Users/oltreseba/Akiflow/akido/node_modules/@electron/universal/src/index.ts:200:13)
    at MacPackager.doPack (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/macPackager.ts:125:9)
    at MacPackager.pack (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/macPackager.ts:179:7)
    at Packager.doBuild (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:441:9)
    at Object.executeFinally (/Users/oltreseba/Akiflow/akido/node_modules/builder-util/src/promise.ts:12:14)
    at Packager._build (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:376:31)
    at Packager.build (/Users/oltreseba/Akiflow/akido/node_modules/app-builder-lib/src/packager.ts:337:12)
    at Object.executeFinally (/Users/oltreseba/Akiflow/akido/node_modules/builder-util/src/promise.ts:12:14)

@oltreseba, @indutny-signal, I fixed it without excluding files. In my package.json, I had custom additions to "files": ... which disables the default inclusions according to the electron-builder documentation. Apparently, the mergeASARs API requires some of those default files that were not in my custom list, so it failed to reconcile. Since the default for "files": ... already included my custom additions, I simply deleted it. Problem fixed. Hopefully, this fixes it for you as well. If so, I'll close this issue.

oltreseba commented 2 years ago

Not really. I'm using files only to exclude additional files in my repository from being build. Also I'm trying to find the default value for files in electron builder, but I cannot find anything. Can you point out the list of default values? - It would be of great help.

Thank you.

indutny-signal commented 2 years ago

@oltreseba you might need to exclude everything from "!**/node_modules/better-sqlite3/build/**" and add back the "node_modules/better-sqlite3/build/Release/better_sqlite3.node",. Even without merging ASARs you really don't want to have these extra files around because they could take a lot of space.

WrathofthePast commented 2 years ago

Not really. I'm using files only to exclude additional files in my repository from being build. Also I'm trying to find the default value for files in electron builder, but I cannot find anything. Can you point out the list of default values? - It would be of great help.

Thank you.

@oltreseba, here's the link to that documentation.

PGIII commented 2 years ago

Having the same issue 

Can't reconcile two non-macho files node_modules/@serialport/bindings-cpp/build/Release/bindings.node  failedTask=build stackTrace=Error: Can't reconcile two non-macho files node_modules/@serialport/bindings-cpp/build/Release/bindings.node

running file on the bindings.node returns this

node_modules/@serialport/bindings-cpp/build/Release/bindings.node: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64Mach-O 64-bit bundle x86_64] [arm64:Mach-O 64-bit bundle arm64Mach-O 64-bit bundle arm64]
node_modules/@serialport/bindings-cpp/build/Release/bindings.node (for architecture x86_64): Mach-O 64-bit bundle x86_64
node_modules/@serialport/bindings-cpp/build/Release/bindings.node (for architecture arm64):  Mach-O 64-bit bundle arm64

Removing custom settings for the build files doesn't seem to change anything

AlithAnar commented 2 years ago

I've got a similar issue, however with node-mac-permissions package:

Can't reconcile two non-macho files node_modules/node-mac-permissions/build/Release/nothing.a  failedTask=build stackTrace=Error: Can't reconcile two non-macho files node_modules/node-mac-permissions/build/Release/nothing.a

I can't ignore this directory in build.files options, because runtime error appears:

Error: Cannot find module 'node-mac-permissions'

Rolling back to older electron-builder version is not an option after upgrading to the latest MacOS

has-n commented 2 years ago

Can confirm the same issue with node-mac-permissions with a small difference.

First trip up is on node_modules/node-mac-permissions/node-addon-api/nothing.target.mk

quanglam2807 commented 2 years ago

Adding this to files in electron-builder config fixes the problem for us: '!node_modules/**/*.{mk,a,o,h}'

It basically ignores all the C/C++ build files from packaging

hrueger commented 1 year ago

I've got a similar issue, however with node-mac-permissions package:

Same with usb-detection. The binary is already universal.

For now, I have to stay with version 22.xx.xxx Python 2 (can be downloaded from here: https://www.python.org/downloads/release/python-2718/)

Don't forget to tell electron-builder where to find it using the PYTHON_PATH Variable:

export PYTHON_PATH=/usr/local/bin/python
yarn electron-builder

Edit 1: it also works with electron-builder v23, as long as you set "mergeASARs": false. However, I'd really like to merge asars...

Edit 2: The binary of usb-detection is not in mach-o format. That is why it does not work. That means, it's not a bug of electron/universal but rather of usb-detection.

Edit 3: It seems like the binary is in Mach-O format, but already contains both architectures:

hannes@macmini node-usb-detection % file build/Release/detection.node
build/Release/detection.node: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bot bundle x86_64Mach-O 64 bit undle x86_64] [arm64:Mach-O 64-bit bundle arm64Mach-O 64-bit bundle arm64]
build/Release/detection.node (for architecture x86_64): Mach-O 64-bit bundle x86_64
build/Release/detection.node (for architecture arm64): Mach-O 64-bit bundle arm64

Seems like electron/universal can't process those binaries?

DaveFlashNL commented 1 year ago

i have a similar issue to this, but with this error:

⨯ Can't reconcile two non-macho files node_modules/dtrace-provider/libusdt/libusdt.a failedTask=build stackTrac

any ideas?

hrueger commented 1 year ago

Adding this to files in electron-builder config fixes the problem for us: '!node_modules/**/*.{mk,a,o,h}'

It basically ignores all the C/C++ build files from packaging

Did you try this? You probably don't need .a files in your app.asar as they are just temporary files of the build process.

Nantris commented 1 year ago

In my case adding '!node_modules/**/*.{mk,a,o,h}' just results in it failing on some other file, and when I got to the end of those it failed in another way.

Detected unique file "node_modules/argon2/bin/darwin-arm64-113" in "/Users/me/Documents/app/release/prod/mac-universal--arm64/app.app/Contents/Resources/app.asar" not covered by allowList rule: "undefined"

and then in Electron-Builder if I add that file to singleArchFiles then I get:

 Command failed: lipo /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node -create -output /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node and /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node have the same architectures (arm64) and can't be in the same fat output file
  failedTask=build stackTrace=Error: Command failed: lipo /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node -create -output /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node and /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node have the same architectures (arm64) and can't be in the same fat output file
mmaietta commented 1 year ago

In this command from the logs above ☝️ 

 Command failed: lipo /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node -create -output /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node

Both paths for lipo are the same. /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node Note the arm64-113/argon2.node in both /T/arm64-mw1MJx and /T/x64-BD57LD dirs This seems like a bug in electron/universal? Why is lipo being used on a single-arch file?

dlon commented 1 year ago

In this command from the logs above point_up

 Command failed: lipo /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node -create -output /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/x64-BD57LD/node_modules/argon2/bin/darwin-arm64-113/argon2.node

Both paths for lipo are the same. /private/var/folders/y5/pp8xq1qd1f78jth_11nsxndh0000gn/T/arm64-mw1MJx/node_modules/argon2/bin/darwin-arm64-113/argon2.node Note the arm64-113/argon2.node in both /T/arm64-mw1MJx and /T/x64-BD57LD dirs This seems like a bug in electron/universal? Why is lipo being used on a single-arch file?

What needs to be done to fix this? We have this same problem when building a universal app on M1 with electron-builder, and we're currently working around the issue by deleting some of the native libraries in beforePack.

Nantris commented 1 year ago

@dlon did you come across this issue? https://github.com/ranisalt/node-argon2/issues/374

I'm not sure this particular issue is with @electron/universal