search

com-lihaoyi / mill

Mill is a fast JVM build tool that supports Java and Scala. 2-4x faster than Gradle and 4-10x faster than Maven for common workflows, Mill aims to make your project’s build process performant, maintainable, and flexible
https://mill-build.org/
MIT License
2.22k stars 356 forks source link

Test `mill.javascriptlib.HelloWorldTests.run` broken locally #3997

Closed myyk closed 2 days ago

myyk commented 2 days ago

Why does this fail locally but not in github test runners?

Not sure, but nothing is getting installed in the node_modules/ when I run the test.

Nothing is even installed when I run the command in npmInstall in an empty directory like this either: npm install --save-dev typescript@5.6.3 @types/node@22.7.8 esbuild@0.24.0

For the node_modules/ to be created and the packages to get installed, I need to first have a package.json file in my directory which is the normal way to use npm.

Repro:

./mill javascriptlib.test

Expected:

That the user can run the test with just mill and npm installed.

Output:

[1071/1071] javascriptlib.test.test
[1071] -------------------------------- Running Tests --------------------------------
[1071] mill.testkit.UnitTester: [6] qux.compile failed
[1071] mill.testkit.UnitTester: [8] qux.run failed
[1071] X mill.javascriptlib.HelloWorldTests.run 1125ms 
[1071]   scala.MatchError: Left(java.io.IOException: Cannot run program "/home/myyk/workspace/mill/myyk/mill/out/javascriptlib/test/test.dest/sandbox/out/mill-test-base-module/930028314041602727/out/qux/npmInstall.dest/node_modules/typescript/bin/tsc" (in directory "/home/myyk/workspace/mill/myyk/mill/out/javascriptlib/test/test.dest/sandbox/out/mill-test-base-module/930028314041602727/out/qux/compile.dest"): error=2, No such file or directory
[1071]       java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
[1071]       java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
[1071]       os.proc.proc$lzycompute$1(ProcessOps.scala:343)
[1071]       os.proc.os$proc$$proc$1(ProcessOps.scala:342)
[1071]       os.proc.spawn(ProcessOps.scala:353)
[1071]       os.proc.spawn(ProcessOps.scala:377)
[1071]       os.proc.call(ProcessOps.scala:222)
[1071]       os.call$.apply(ProcessOps.scala:42)
[1071]       mill.javascriptlib.TypeScriptModule.$anonfun$compile$4(TypeScriptModule.scala:54)
[1071]   java.io.IOException: error=2, No such file or directory
[1071]       java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
[1071]       java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
[1071]       java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
[1071]       java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
[1071]       java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
[1071]       os.proc.proc$lzycompute$1(ProcessOps.scala:343)
[1071]       os.proc.os$proc$$proc$1(ProcessOps.scala:342)
[1071]       os.proc.spawn(ProcessOps.scala:353)
[1071]       os.proc.spawn(ProcessOps.scala:377)
[1071]       os.proc.call(ProcessOps.scala:222)
[1071]       os.call$.apply(ProcessOps.scala:42)
[1071]       mill.javascriptlib.TypeScriptModule.$anonfun$compile$4(TypeScriptModule.scala:54)) (of class scala.util.Left)
[1071]     mill.javascriptlib.HelloWorldTests$.$anonfun$tests$2(HelloWorldTests.scala:26)
[1071] Tests: 1, Passed: 0, Failed: 1
[1071/1071] ========================================================================= javascriptlib.test ============================================================================== 4s
myyk commented 2 days ago

Oh, I found where npm install is creating a package.json and /node_modules/, it's in my ~ directory. I've noticed that getting cluttered recently, probably from running the mill test suite. This feels wrong to me.

myyk commented 2 days ago

@lihaoyi I tried a few things to fix this and none of them worked.

But at a high level, I'm thinking...

How should Mill cooperate with npm or other JS/TS build tools? Also when is it useful to use mill instead of those tools?

For the first one though, I struggle to think how useful one could use npm without managing a package.json. I know it sucks to then have two build tools, but I would have to assume the user has their reasons. By not using one, the user also has to give up a lot of the rest of the npm ecosystem which is kind of a false dichotomy when mill is using npm to build.

Some reasons to use a package.json/package-lock.json:

It would be a large effort to have mill implement feature parity with npm, but it might still be useful sometimes for mill to tie together npm commands to get something larger done.

lihaoyi commented 2 days ago

On my laptop (OS-X, M1 Max, Java 17, NPM 8.19.3, Node 18.13.0), on a clean latest checkout, that command seems to pass. 

$ ./mill javascriptlib.test
...
[1071/1071] javascriptlib.test.test
[1071] -------------------------------- Running Tests --------------------------------
[1071] + mill.javascriptlib.HelloWorldTests.run 314627ms
[1071] Tests: 1, Passed: 1, Failed: 0

What OS/CPU/Java/NPM/Node versions are you running? Maybe there's some combination that is incompatible.

In particular, it seems suspicious if Mill is installing stuff in your home directory. After all, npm does not install stuff globally without -g/--global, which we don't pass as part of the TypescriptModule integration. Are you sure the stuff in home isn't from other projects? Do you have some .npmrc or some other global NPM config that could be affecting this?

The integration between Mill and other tools is something we'll need to explore as part of this exercise. I guess at a high level, you would want to use Mill in two scenarios:

  1. If your project has multiple "modules", and you want to benefit from incremental caching and parallelism at a module level.

    • e.g. if your 5mLOC typescript typecheck takes 30 minutes (like it does at some of the big companies i talked to), and tsc does not have incremental re-compiles, you may want to break down that 5mLOC monolith into 50 100kLOC modules with 30-40s incremental re-typechecks (at a module granularity) and <30min full re-compiles (assuming some modules can be typechecked in parallel)

  2. If your project has multiple languages or language toolchains, and you want to wire the up together.

    • One obvious example is you have a Java/Scala/Kotlin webserver with a Typescript frontend client, maybe with some custom static resource pre-processing for both the server and client (e.g. appending content-address SHA suffices to every resource for cache busting).
    • Using either a purely-JVM toolchain (maven, gradle) or purely-JS toolchain (NPM, Webpack) may be unfeasible, using both JVM and JS toolchains together and wiring them up with bash scripts may be awkward, especially if it grows (what if you have 5 JVM servers and 5 JS frontends with some shared code?). That would be the scenario you may want to use Mill

Ultimately Mill will need to integrate with NPM/Yarn/TSC/etc. to do the actual work, just like how on the JVM we need to integrate with Coursier/Zinc/etc. There will definitely be some awkwardness in the integration and some duplication of concepts across the different tools. But the reason you would want to use Mill is if the above two scenarios are sufficiently painful that the benefits outweigh the integration cost

myyk commented 2 days ago

@lihaoyi thank you! I did have a ~/.npmrc that I literally have no idea how or when it got there, I've had this OS installed for a while now. It contained: prefix=/home/myyk/.npm-global

This was probably not version related but In case this helps someone searching in the future:

$ node --version
v20.17.0
$ npm --version
10.8.2
$ uname -a
Linux DevBox 6.8.0-48-generic #48-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 14:04:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
$ java --version
openjdk 17.0.13 2024-10-15
OpenJDK Runtime Environment (build 17.0.13+11-Ubuntu-2ubuntu124.04)
OpenJDK 64-Bit Server VM (build 17.0.13+11-Ubuntu-2ubuntu124.04, mixed mode, sharing)

That seems to have been enough to make everything not work for me. The tests now pass for me!

As for use cases, that is pretty much what I was thinking. It's nice to hear you've found some potential companies that might use this already.

I'm interested to work more on how mill and NPM/Yarn/TSC/etc play together. I'll find another channel to discuss how to develop those interactions more since I think this bug was a bug in my computer.