Find npm in wrong place

[rhdtl78]

Related the document, plugin have to find npm on global scope, but it is not.

Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command 'npm'' at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241) ... org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) ... Caused by: java.io.IOException: Cannot run program "npm" (in directory "~/projects/projectRoot"): error=2, No such file or directory

So, I tried to override npm or node excitable path but no effect

My environment is like below.

os.arch = aarch64 os.name = Mac OS X os.version = 11.2.2 openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment Zulu11.45+27-CA (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM Zulu11.45+27-CA (build 11.0.10+9-LTS, mixed mode)

node -v v15.10.0 process.arch: arm64

[bsautel]

I confirm that by default the plugin uses the npm command available in the PATH and it is how I use it. The integration tests ensure that we do use the globally or locally installed node and npm commands according to the configuration and they run on both Windows, macOS and Linux. You are probably in a really specific case.

You will probably think that my question is stupid but I ask it anyway: is npm available in the PATH in your environment?

[rhdtl78]

I confirm that by default the plugin uses the npm command available in the PATH and it is how I use it. The integration tests ensure that we do use the globally or locally installed node and npm commands according to the configuration and they run on both Windows, macOS and Linux. You are probably in a really specific case.

You will probably think that my question is stupid but I ask it anyway: is npm available in the PATH in your environment?

Sorry for late reply.

I'm using node installed by nvm. I can us node globally, and nvm node path is also exported.

[bsautel]

Ok for node, but is npm also available in the PATH. If you run it in the same environment (terminal) directly or via Gradle, it should work. Are you sure you are using the same environment when running Gradle?

[rhdtl78]

Yeah. I'm sure it is same env. Maybe it caused by architecture. I can build project by Gradle with this plugin on Rosetta2, but not in M1 Native.

Maybe this issue related with this

[bsautel]

Indeed I did not noticed you are running a Mac M1. But I don't see why the npm command works when you run it from the terminal and not when it is launched by Gradle.

Can someone using a Mac M1 confirm that it the problem comes from the platform?

The issue #154 is not exactly the same problem. You are not asking the plugin to download and install Node.js, and this is what was reported as broken on Mac M1.

[rhdtl78]

Sorry for late replay.

Out of curiosity, if you run this with a x64 JVM does it work?

Yeah. I can run on x86 JVM OpenJDK 11 with rosetta2. Maybe this is the workaround for now. You can close this issue.

[bsautel]

Ok, so you mean that the issue happens only when running an arm64 JVM. If you run a x86 one using Rosetta 2, you don't encounter the issue, right?

[rhdtl78]

exactly.

[bsautel]

Ok, that's interesting. We use the Gradle exec API to execute node, npm and yarn. So either the issue is in the Gradle exec API implementation but I assume this is more probable that it comes from an issue in the arm64 JVM.

Can you run this class using the x64 JVM and the arm one?

import java.io.IOException;

import static java.nio.charset.StandardCharsets.UTF_8;

public class JavaNpmExec {
    public static void main(String[] args) throws IOException {
        String command = "npm -version";
        Process child = Runtime.getRuntime().exec(command);
        System.out.println(new String(child.getInputStream().readAllBytes(), UTF_8));
        System.err.println(new String(child.getErrorStream().readAllBytes(), UTF_8));
    }
}

If you use Java 11+, you can simply create a JavaNpmExec.java file containing this anywhere on your filesystem and run it without compiling it by using java JavaNpmExec.java.

It runs npm which is expected to be available in the PATH (add it if it is not). It should work with the x64 JVM and probably not with the arm one.

[alexpartsch]

@bsautel I'm running into the same issue when having download = false set on a Apple M1 Book, running the given Java program with aarch64 on JDK 16:

/tmp % java JavaNpmExec.java
7.6.0

Using a x86 compatible JDK 16 on M1 yields the same:

/tmp % java /tmp/JavaNpmExec.java
7.6.0

[bsautel]

Ok, thanks for your feedback @alexpartsch. Unfortunately this experiment did not enable us to spot the origin of the issue but that's quite a good news there is not a such big bug in the arm64 JVM!

We still have to find where does the issue come from. Let's check something on the Gradle side.

Gradle 7.0-rc1 was released a few days ago and adds full support of arm64 JVM. They don't talk about exec related issues in the release notes but could someone check whether the issue is still here when using an arm64 JVM with Gradle 7.0-rc1 please?

[michal-kaciuba]

Hi! Got the same issue on M1. Using Gradle v7.2 and gradle-node-plugin v3.2.1.

[deepy]

@MKaciuba just to confirm, that's getting Cannot run program "npm" (in directory ... when running with download = false right?

[lonely-lockley]

Faced this issue on Intel Mac. The funny thing that it stopped working without any external changes. The day before I've launched several builds successfully and today 'Cannot run program "npm"'. Everything works fine from terminal. Liberica JDK 21, Gradle 8.5, Ventura 13.2.1, plugin version 7.0.1

[lonely-lockley]

Could workaround the problem by setting:

node {
    npmCommand = '/usr/local/bin/npm'
}

Adding println System.getenv('PATH') to build.gradle showed that /usr/local/bin IS IN the PATH so the real cause of the problem is still not clear

[deepy]

Are you using download = true or should it run with local node? And which task is failing?

Assuming it's download = true, can you run your build with --info and see whether nodeSetup and npmSetup reports up-to-date, check in the .gradle folder in the project and see which versions exists under nodejs and npm Then run ./gradlew nodeSetup npmSetup --rerun-tasks and if the build works after that, see if any new versions appeared under the nodejs and npm folders

[lonely-lockley]

Download is set to false by default. I tried to run npmInstall

[tzcsx]

I am also experiencing this:

Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'npm'
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
        at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
        at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:122)
        at org.gradle.process.internal.ExecHandleRunner.lambda$run$0(ExecHandleRunner.java:80)
        at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:80)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:79)
        ... 2 more
Caused by: java.io.IOException: Cannot run program "npm" (in directory "/Users/me/dev/project/frontend"): error=2, No such file or directory
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)

Reproducible for the first build run via IntelliJ after system boot. Once this error occurs, it persists for builds outside of IntelliJ, too. Stopping all Gradle daemons with ./gradlew --stop and starting a new build fixes the problem until the next reboot.

I can confirm that Gradle itself sees the correct $PATH, which includes npm, but having poked around in the source I am not sure if NpmExecRunner uses the same path. IntelliJ also sees the correct $PATH, so it should not be related to the shell environment loading https://intellij-support.jetbrains.com/hc/en-us/articles/15268184143890-Shell-Environment-Loading.

When using

node {
  npmCommand = "/Users/me/.nvm/versions/node/v20.11.0/bin/npm"
}

this issue does not occur at all.

Environment

download = false (using default)
Node & NPM via nvm
macOS 14.2 on M2
Java 21.0.2
Gradle 8.6
IntelliJ 2023.3.5

[tzcsx]

Correction: Every build started via IntelliJ UI will fail with this error, if this build creates a new Gradle Daemon instance. If there is a Daemon created by a build outside of IntelliJ, i.e. terminal, the build run from IntelliJ will work, too. However directly triggering npm from IntelliJ UI works and a terminal started inside IntelliJ sees npm, too. So I still think this is unrelated to IntelliJ shell environment loading.

I am unsure whether the Gradle version is a factor. Still investigating.

[deepy]

@tzcsx ARM mac? Can you add the output of sysctl sysctl.proc_translated when run through the failing scenario?

var process = new ProcessBuilder("sysctl", "sysctl.proc_translated").inheritIO().start().waitFor();

I suspect you're hitting the issue at: https://github.com/node-gradle/gradle-node-plugin/issues/154#issuecomment-1825542642

[tzcsx]

Yes, ARM mac M2 and download = false. I tried this:

tasks.register<NpmTask>("doStuffWithNpm") {
    args.add("do-stuff")

    val process = ProcessBuilder("sysctl", "sysctl.proc_translated").start()
    val output = BufferedReader(InputStreamReader(process.inputStream))
    process.waitFor()
    println("configure " + System.getProperty("os.arch") + " " + output.readLine());

    doFirst {
        val process = ProcessBuilder("sysctl", "sysctl.proc_translated").start()
        val output = BufferedReader(InputStreamReader(process.inputStream))
        process.waitFor()
        println("doFirst " + System.getProperty("os.arch") + " " + output.readLine());
    }
}

This results in "_aarch64 sysctl.proctranslated: 0" for all phases in both the running and failing scenarios.

[DylanLukes]

+1 to experiencing this (and also on an M1 Mac), though my npm is at /Users/dylan/.asdf/shims/npm. Works perfectly fine when running gradle from the command line. Inspecting the state of the ProcessBuilder at the point of failure, I can see that /Users/dylan/.asdf/shims/ is indeed on the path, so the shell environment not being present is certainly not the issue.

Even stranger, if I pause the debugger at the point the exception is thrown and run:

new String(new ProcessBuilder("npm", "--version").start().getInputStream().readAllBytes());

...it spits back out the correct version, apparently able to find npm with no problem!

---

Also, for me this produces 0 in the failing case, and I'm not downloading any npm.

ProcessBuilder("sysctl", "sysctl.proc_translated").inheritIO().start().waitFor();

[DylanLukes]

The last handoff point is in executeCommand in NpmExecRunner:

https://github.com/node-gradle/gradle-node-plugin/blob/92e1d218789d9c69266ad58328a6a698e80ec94e/src/main/kotlin/com/github/gradle/node/npm/exec/NpmExecRunner.kt#L39-L62

But the ExecConfiguration produced there doesn't seem too unusual, except for the additionalBinPaths being spurious:

execConfiguration = {ExecConfiguration@19886} ExecConfiguration(executable=npm, args=[install], additionalBinPaths=[/Users/dylan/IntelliJIDEAProjects/tilia/projects/service/www/.gradle/nodejs/node-v18.17.1-darwin-arm64/bin, /Users/dylan/IntelliJIDEAProjects/tilia/projects/service/www/.gradle/nodejs/node-v18.17.1-darwin-arm64/bin], environment={}, workingDir=null, ignoreExitValue=false, execOverrides=null)
 executable = "npm"
 args = {ArrayList@19897}  size = 1
 additionalBinPaths = {ArrayList@19898}  size = 2
  0 = "/Users/dylan/IntelliJIDEAProjects/tilia/projects/service/www/.gradle/nodejs/node-v18.17.1-darwin-arm64/bin"
  1 = "/Users/dylan/IntelliJIDEAProjects/tilia/projects/service/www/.gradle/nodejs/node-v18.17.1-darwin-arm64/bin"
 environment = {RegularImmutableMap@19836}  size = 0
 workingDir = null
 ignoreExitValue = false
 execOverrides = null

The additionalBinPaths here don't seem be an issue as if I delete them before allowing execution to resume, and this configuration is identical across all configurations (command line, IDEA, IDEA launched from command line).

---

Manually running a simplified case in the same execution context seems to produce the same issue.

execRunner.execute(project, extension, ExecConfiguration("npm", listOf("--version")))

However, this works:

ProcessBuilder("npm", "--version").start().inputReader().readText()

[DylanLukes]

Well, that's interesting. I get this failure when I run IntelliJ by opening it as an app, but if I start IntelliJ from a shell then I have no issues.

cd project-dir
idea .

Not exactly a satisfying workaround, but might point to the root of the issue (in conjunction with it not being a PATH issue...)

Part of what's still bugging me is that this is only happening for npm and not say for uname which runs through the same process launcher ahead of npm. Maybe there's some MacOS security system related thing going on, the difference being one is a trusted system binary and the other isn't...? Running from the command line could confer some entitlements or relaxation of security. But then why does a ProcessBuilder like at the end of my prior comment work just fine?

[DylanLukes]

Well, this is interesting...

So, the issue doesn't seem to be finding npm at all.

It appears that setting environment to null here allows execution to proceed normally, though merely emptying it does not.

It's not clear to me exactly why yet, but I've traced it all the way down to a native call (forkAndExec) in the constructor of java.lang.ProcessImpl:

    private ProcessImpl(final byte[] prog,
                final byte[] argBlock, final int argc,
                final byte[] envBlock, final int envc,
                final byte[] dir,
                final int[] fds,
                final boolean forceNullOutputStream,
                final boolean redirectErrorStream)
            throws IOException {

        pid = forkAndExec(launchMechanism.ordinal() + 1,
                          helperpath,
                          prog,
                          argBlock, argc,
                          envBlock, envc,    // <- nulling out envBlock right before call results in success
                          dir,
                          fds,
                          redirectErrorStream);
        processHandle = ProcessHandleImpl.getInternal(pid);

        try {
            AccessController.doPrivileged((PrivilegedExceptionAction<Void>) () -> {
                initStreams(fds, forceNullOutputStream);
                return null;
            });
        } catch (PrivilegedActionException ex) {
            throw (IOException) ex.getCause();
        }
    }

As one might expect, setting the envBlock to null here results in success running npm and the npmInstall task succeeding without a hitch.

I've got a diff of the environments running IntelliJ normally (from GUI) and with the idea command from within a terminal and tried splicing changes in to see if anything clicks, but of course it doesn't. It really seems like the issue stems from having environment set at all. Even an empty environment in the ProcessBuilder fails. It's null or no go.

This is probably something to do with macOS permission weirdness. Who knows, it might just suddenly start working after an update... still, that a null environment seemingly gets it to work is a bit bizarre.

[DylanLukes]

Well, that's out too. In the debugger, at the site of processBuilder.start(), running:

new String(new ProcessBuilder(processBuilder.command).directory(processBuilder.directory).start().getInputStream().readAllBytes())

Produces:

added 157 packages, and audited 158 packages in 588ms

42 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

The only difference between this process builder and the one created by Gradle internals is the explicitly set environment, nothing else. And nulling that environment back to its default state in that one allows execution to continue. I think this is fully ablated, I'm just not sure what to make of the result:

• Why would explicitly setting the environment behave differently when the Gradle process is running under IntelliJ vs under shell?
• Why would it matter if I start IntelliJ from the shell? The idea launcher doesn't own the resulting process, launchd does.
• Why can npm be found when the environment is inherited, but not when it's explicit set to exactly the same dict (I checked)?

[DylanLukes]

Well, in the end I think this issue is not at all related to gradle-node-plugin in particular, and couldn't be fixed here. In an empty project:

tasks.register("mystery") {
    val pb = ProcessBuilder("lua", "--version")
    pb.environment().clear()
    pb.environment().putAll(System.getenv())
    val str = pb.start().inputStream.readAllBytes().decodeToString()
    println(str)
}

This also fails (lua is not an asdf shim but also not on the default path). Things that are on the default system path do work, such as env... which reports back the full PATH I would expect, so I'm flummoxed.

[tzcsx]

I am still running into this every day. The first build of the day needs to be started outside of Intellij in order to produce a working Gradle Daemon process which successive Intellij builds will then use.

@DylanLukes You've dug deep there, thanks. So it looks like this is either a Gradle or Intellij issue, but nothing @deepy can look into. Could you escalate this in a Gradle ticket? The Gradle team should be able to decide if this is related to Gradle itself or if it is a problem with Intellij's MacOS init logic.

[deepy]

Following the investigation I think this sounds like it could be related to https://github.com/gradle/gradle/issues/10483#issuecomment-2141505147 and https://youtrack.jetbrains.com/issue/IDEA-334183

Which to my understanding means that if you're on Java 21 PATH changes won't work and the ideal workaround right now would be starting IDEA from your shell

i.e. PATH changes will look like they work, but won't

In short this is a bit of a headache and means I need to run cross-version tests for Java as well Which I guess means a new test suite as the serial time on a beefy machine is right now 3 hours

[tzcsx]

@deepy Good find! I can confirm that either building with Java 17 instead of Java 21 or using Java 21 and starting Intellij from the Terminal via open -a IntelliJ\ IDEA will work.