Support for Apple Silicons architecture aarch64/arm64

[fkfhain]

When building modules using gradle-node-plugin I do get:

Could not find org.nodejs:node:12.18.4. Searched in the following locations:

• https://jcenter.bintray.com/org/nodejs/node/12.18.4/node-12.18.4.pom
• https://repo.maven.apache.org/maven2/org/nodejs/node/12.18.4/node-12.18.4.pom
• https://nodejs.org/dist/v12.18.4/node-v12.18.4-darwin-arm64.tar.gz

Enviroment:

• gradle version 6.8.1
• gradle-node-plugin 3.0.1
• OpenJDK Runtime Environment Zulu11.45+27-CA (build 11.0.10+9-LTS)

Any plans to incorporate aarch64/arm64 support in upcoming releases? Or Any ideas on how to enforce a download for ~x64 versions of node (which will run in rosetta2-comp mode)?

Thanks a lot.

[deepy]

the best solution I can think of right now is adding a configurable override on what's to be fetched, that'd mean you'd have to write the logic for determining what to pick in your build script.

[mshima]

We are considering enabling download by default at jhipster. But this is a blocker IMO.

It should fetch amd64 instead of arm64. There won’t be a stable arm64 so soon.

[fkfhain]

the best solution I can think of right now is adding a configurable override on what's to be fetched, that'd mean you'd have to write the logic for determining what to pick in your build script.

That would be lovely .. Something like a fullPathToResource that we would set as follows

node {
    version.set("12.18.4")
    npmVersion.set("6.13.7")
    download.set(true)
fullPathToResource.set("https://nodejs.org/dist/v12.18.4/node-v12.18.4-darwin-x64.tar.gz")
}

depending on architecture ..

or to hacky?

[deepy]

@fkfhain I'm thinking something more along the lines of

node {
    nodeArchitectureOverride.set({
        if (Os.isFamily(Os.FAMILY_MAC)) {
            return "darwin-x64"
        }
        return null
    })
}

[bsautel]

Just to be sure I understand what is happening here. We detect that the system is a macOS running on a arm architecture but the corresponding Node.js bundle does not exist yet so the download fails. The workaround is to install the x64 version that is supported by Apple Silicon system thanks to the emulator, but we cannot add that to the configuration. Am I right?

If this is true, should not we take this into consideration when determining the platform for which we download the bundle?

[deepy]

The only two arguments I can think of going against it is that: 1. it's work and 2. if we don't offer a way to override that using a custom repo which does contain a node built for arm-Macs we'd have no way of making use of that.

[bsautel]

Is it more complicated than returning darwin-x64 if OS os darwin and platform armxxx (I don't know what the exact arch code is for Apple M1 systems)?

You mean you would want to make it possible to override it for people using custom Apple Silicon builds?

I don't know this topic very well, but it sounds like we should handle this specificity on our side in order to support Apple Silicon systems out of the box. When the Apple Silicon build is available, we will be able to adapt the detection mechanism to use the right architecture if available. For that we could use the Node.js version. For instance if we assume it will be supported in Node.js 16 and not below, we could state that we use the x64 bundle for Node.js < 16 and the arm bundle for Node.js >= 16. What do you think about that?

[deepy]

I tried messing around with dependencySubstitutions to see if it's possible to work around this from the user's end. Couldn't get it to work.

We should just handle it on our end

[fkfhain]

Thanks for looking into this. Highly appreciated.

Just to clarify: The problem is that the construction of the download url ex https://nodejs.org/dist/v12.18.4/node-v12.18.4-darwin-arm64.tar.gz is absolutely correct. At the end it is a Mac (darwin) running on ARM (although .. the correct arch type might be something like "aarch64").

Obviously those packages (darwin + mac) don't exist for all the (older) nodejs versions out there. So .. 404. But .. utilizing Apples rosetta-2 emulation on the fly black magic .. its darwin-x64 counter part would work.

Hence the suggestion to provide an override-url that downloads and attempts to execute the packaged regardless of recognized architecture on the build client.

[bsautel]

Can someone tell us what the os.arch Java property contains when running Java on macOS using a M1 CPU?

I am pretty sure that we can fix that quite easily by forcing to return darwin-x64 instead of darwing-armwhatever.

[fkfhain]

os.arch -> aarch64 on MBA with M1

But overriding this on a global level definitely has probably undesired side-effects ..

[bsautel]

I changed the platform detection behavior to return x64 when running on a Mac with aarch64 architecture in the apple-silicon branch. Could someone with a Mac M1 test that please?

Here is how to test that. First, checkout this repository, switch to the apple-silicon branch. Then:

1. Either you run the plugin's tests, it contains some integration tests that will download Node.js, but it can be quite long.
2. Or build a project that uses the plugin and is configured to download Node.js (and thus should fail with the current version) with these additional Gradle parameters: --include-build ../path/to/node-gradle-plugin. This will ask Gradle to use the plugin from the source code instead of from the plugins repository.

Hope this will work! I don't think that this will have undesired side effects since the code I modified is only used to determine which Node.js bundle to use.

[fkfhain]

@bsautel .. I did both. Mixed results: As for 2.) .. full success. A build with --include-build [path] runs like charm. Thanks!

As for 1.) The build fails with a large number of variations of

NpmInstall_integTest > install packages with npm[1] FAILED
    org.gradle.testkit.runner.UnexpectedBuildFailure: Unexpected build execution failure in /private/var/folders/f8/72vh7st17dg462x3fxrp_5mm0000gn/T/junit15186718974381630875 with arguments [npmInstall, --warning-mode=fail]

    Output:
    > Task :nodeSetup SKIPPED
    > Task :npmSetup SKIPPED
    > Task :npmInstall FAILED

    FAILURE: Build failed with an exception.

When running with --info a warning is printed

WARNING: TestExecutionListener [org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener] threw exception for method: executionFinished(TestIdentifier [uniqueId = '[engine:junit-vintage]/[runner:com.github.gradle.node.util.PlatformHelperTest]/[dynamic:verify ARM handling aarch64 (arm64)(com.github.gradle.node.util.PlatformHelperTest)]', parentId = '[engine:junit-vintage]/[runner:com.github.gradle.node.util.PlatformHelperTest]', displayName = 'verify ARM handling aarch64 (arm64)', legacyReportingName = 'verify ARM handling aarch64 (arm64)', source = ClassSource [className = 'com.github.gradle.node.util.PlatformHelperTest', filePosition = null], tags = [], type = TEST], TestExecutionResult [status = SUCCESSFUL, throwable = null])
java.lang.AssertionError
    at org.gradle.api.internal.tasks.testing.processors.TestOutputRedirector.setOutputOwner(TestOutputRedirector.java:49)
    at org.gradle.api.internal.tasks.testing.processors.CaptureTestOutputTestResultProcessor.completed(CaptureTestOutputTestResultProcessor.java:80)
    at org.gradle.api.internal.tasks.testing.results.AttachParentTestResultProcessor.completed(AttachParentTestResultProcessor.java:56)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.actor.internal.DefaultActorFactory$BlockingActor.dispatch(DefaultActorFactory.java:128)
    at org.gradle.internal.actor.internal.DefaultActorFactory$BlockingActor.dispatch(DefaultActorFactory.java:100)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at com.sun.proxy.$Proxy6.completed(Unknown Source)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener.executionFinished(JUnitPlatformTestExecutionListener.java:108)
    at org.junit.platform.launcher.core.TestExecutionListenerRegistry$CompositeTestExecutionListener.lambda$executionFinished$10(TestExecutionListenerRegistry.java:109)
    at org.junit.platform.launcher.core.TestExecutionListenerRegistry.lambda$notifyEach$1(TestExecutionListenerRegistry.java:67)

But I'm unsure if that warning is related ..

[bsautel]

Thanks for you quick answer.

Our integration tests are unfortunately very unstable. That sometimes the case on my laptop but it happens much more frequently in GitHub Actions. That's really painful, it's the reason why we retry multiple times the failing tests. We can see tests failures in the terminal but the build does not fail.

I don't know why, when I run some Node, npm and yarn commands directly or via the Gradle plugin in production, it always work. But in the context of tests, sometimes the commands fail, I don't know really why.

The first error you have is probably an integration test that failed, that has nothing to do with this issue. The second one seems to be related to what I changed. It sounds like it is a warning indicating that a test failed, but it is not a Gradle test failure report. On my side, all the tests are passing, and it is also the case in GitHub Actions on Windows, macOS and Linux. I'll have a look at that but the fact that the tests passed and your real life project test also succeeded seem to show that it fixed the issue.

@deepy what do you think of this fix proposal?

[seregamorph]

@bsautel I've tried option with --include-build on the branch apple-silicon. The nodejs artifact is now resolved as https://nodejs.org/dist/v12.18.2/node-v12.18.2-darwin-x64.tar.gz, but I have another failure now:

internal/modules/cjs/loader.js:834
  throw err;
  ^

Error: Cannot find module '../models/config'
Require stack:
- /Users/morph/Projects/acme/acme-codeserver-framework/codeserver-framework/framework-ui/node_modules/.bin/ng
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:831:15)
    at Function.Module._load (internal/modules/cjs/loader.js:687:27)
    at Module.require (internal/modules/cjs/loader.js:903:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/Users/morph/Projects/acme/acme-codeserver-framework/codeserver-framework/framework-ui/node_modules/.bin/ng:8:19)
    at Module._compile (internal/modules/cjs/loader.js:1015:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1035:10)
    at Module.load (internal/modules/cjs/loader.js:879:32)
    at Function.Module._load (internal/modules/cjs/loader.js:724:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/morph/Projects/acme/acme-codeserver-framework/codeserver-framework/framework-ui/node_modules/.bin/ng'
  ]
}
error Command failed with exit code 1.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':codeserver-framework:framework-ui:buildUi'.
> Process 'command '/Users/morph/Projects/acme/acme-codeserver-framework/codeserver-framework/framework-ui/.gradle/yarn/yarn-v1.22.10/bin/yarn'' finished with non-zero exit value 1

It says, that it cannot find '../models/config', but it exists:

ls codeserver-framework/framework-ui/node_modules/@angular/cli/models/config
config.d.ts
config.js
config.js.map
spec-schema.json

I have an assumption, that for yarn project the relative module path is resolved wrong in M1 for some reason (same build is success on Intel chip mac). Please let me know if you need more details or you'd like to have a separate issue for it, thanks.

[bsautel]

Thanks @seregamorph for your confirmation that the download now works and for the working directory issue report.

It don't really see why an architecture change would lead to break this kind of thing but the issue you are reporting is quite similar to #152 in which npm is in the PATH but the Gradle plugin cannot find it when running a NpmTask and this also happens only on Mac M1.

Could you try something to check whether the issue comes from a wrong working directory?

Add a task to your project like this (Groovy DSL, ask me if you need some help to convert it to Kotlin if your build is in Kotlin)?

task env(type: NodeTask) {
    script = file("env.js")
    outputs.upToDateWhen {
        true
    }
}

Create a env.js file containing that:

console.log(`Current working directory: ${process.cwd()}`);

Then run the env task and check that the current working directory printed corresponds to the location of your project.

[seregamorph]

@bsautel

Project root (multimodule): /Users/morph/Projects/acme/acme-codeserver-framework

Module (with new task): /Users/morph/Projects/acme/acme-codeserver-framework/codeserver-framework/framework-ui/build.gradle

env.js location: /Users/morph/Projects/acme/acme-codeserver-framework/codeserver-framework/framework-ui/env.js

Command: ./gradlew clean :codeserver-framework:framework-ui:env --include-build /Users/morph/Projects/3rdparty/gradle-node-plugin -d

Result contains: 2021-03-24T09:24:57.309+0300 [QUIET] [system.out] Current working directory: /Users/morph/Projects/acme/acme-codeserver-framework/codeserver-framework/framework-ui

(if the env.js file is located in other directory, the build fails)

[bsautel]

The current working directory variable seems to be right. 🤔

In issue #152 we discovered that the issue related to the PATH commands that does not seem to be found only happens when using the arm64 JVM and not the x64 one using Rosetta2. Which JVM are you running? If running the arm one, could you test with the x64 emulated one to see whether the issue is still present?

[deepy]

If anyone works at a company that makes obscene amount of monies this is the perfect time to sponsor either/both of us with a M1 Mac ;-)

Or perhaps with some M1-as-a-Service that we can add to our CI pipeline like the one from scaleway, or just SSH access to a M1 mac

[ghost]

@bsautel the same result (to ensure, I've stopped the daemon first):

echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home

P.S. @seregamorph is my second (personal) account

[bsautel]

Ok. And what if you manually run the command the Gradle plugin is supposed to run. By default the working directory is the one of the project (not the root project). Does it run?

Sorry for all these questions, but I am trying to identify the cause of this issue and I don't have any Mac M1 computer.

[seregamorph]

@bsautel , @deepy I've sent you an e-mail (found your address in git commits), please check, maybe it's in the spam box. Does it work for you?

[deepy]

@seregamorph on my gmail I only have oddly specific targeted spam and I can't see anything in the main inbox If it was sent to my work email it probably won't arrive, that thing is even suspicious of internal emails

[seregamorph]

@deepy okay, whatever. Can you please contact me to serega.morph[at]gmail.com, I'm ready to suggest some options regarding rent M1 machines. I'd like not to discuss it here.

[seregamorph]

I still do not see any answer on my e-mail, probably the spam filter is stubborn (please check your spam folder, maybe you'll find smth interesting :D ) So, I just wanted to say that I'm ready to donate up to 20 eur to the PayPal account (mentioned in readme) for the purposes of renting smth like Scaleway machine.

[bsautel]

@seregamorph thanks for your great proposition! I did not have time to reply before. I replied to your email. @deepy it sounds like it was sent to your personal gmail address.

[HannesOlszewski]

FYI for anyone finding this issue and who can afford to use node v16: The newly released node 16.0.0 provides an official distribution for darwin-arm64 which does work with this plugin. Sadly the releases prior to v16 still don't provide any yet.

[JnManso]

Temporary Fix: Duplicate your terminal in the finder app. Open the info and enable the rosetta. Run the "arch" command to confirm that you are running in i386. Install the brew app and then using brew install the adoptopenjdk11 for example. Run the "java -XshowSettings:properties -version" command and confirm that the "os.arch" is "x86_64". Now compile your app and the gradle node plugin will use the "x86_64" arch.

[deepy]

I think we might want to change the default node version to 16 (even though it's not LTS) and maybe log a warning when using download = true and a version < 16

Thanks for the workaround JnManso, that's going to help a lot of people who can't upgrade :-)

[davinkevin]

The solution proposed doesn't work with tools like asdf 😓. I'm still looking for simple solution for this 😇

[davinkevin]

Found it!

I run my task with the following parameter, to bypass the current os arch. I know it's not optimal for a lot of other things… but 🤷, I can't do anything else, the front is stuck in old node… and especially npm version.

./gradlew frontend-angular:build -Dos.arch=64

[deepy]

There's a general issue for asdf node not working #252 But I'm not entirely certain it's something that can be supported from the plugin's side

As for a better way of doing this, #234 gets easier with every refactor I do and configuration-cache compatible PRs will be accepted

[deepy]

class Process {
    public static void main(String[] args) throws Exception {
        var process = new ProcessBuilder("sysctl", "sysctl.proc_translated").inheritIO().start().waitFor();
        System.out.println(System.getProperty("os.arch"));
    }
}

» arch -arm64 /usr/bin/java Process
sysctl.proc_translated: 0
aarch64
» arch -x86_64 /usr/bin/java Process
sysctl.proc_translated: 1
aarch64

And the underlying issue being that os.arch is of course just talking about the JVM, and with macOS having universal binaries with different architectures you can of course end up in a situation where you're running an "ARM" JDK under Rosetta 🥲

» file /usr/bin/java
/usr/bin/java: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/java (for architecture x86_64):    Mach-O 64-bit executable x86_64
/usr/bin/java (for architecture arm64e):    Mach-O 64-bit executable arm64e