Watch mode on multiple commands should only run commands downstream of changed files (1000USD Bounty)

[nafg]

---

From the maintainer Li Haoyi: I'm putting a 1000USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

The goal of this is to make -w/--watch only run tasks/commands that are downstream of changed Task.Source files or Task.Input, so that if you watch multiple commands (e.g. webserver.runBackground and webclient.deploy), only the tasks relevant to a change get re-run.

---

Background

I think my use case is a pretty common need. I'm developing in full-stack Scala: some JVM modules, some ScalaJS modules, and some shared dependencies of both. I want a fast feedback loop. When code in a JS or shared module changes, it needs to rebundle it, and when code in a JVM or shared module changes, it needs to recompile it and restart the server.

I don't need hot reload of the javascript code (that would be even better but let's get to first base first), but I do need to reload the page after rebundling JS or rebooting the server completes, but I'm accomplishing that outside of the build tool so let's ignore that aspect of things.

In my case, the target to bundle the JS and put it in the right place is app_js.getScalaJs, and restarting the backend is done with app_lrbcol.runBackground.

I've been doing this with two separate Mill processes. tmuxp is useful for this; my config contained

  panes:
  - mill -j 16 -w app_lrbcol.runBackground
  - docker compose up db
  - sleep 5; mill -j 16 -w app_js.getScalaJs

However, at the moment Mill isn't designed for multiple processes like this (see e.g., #3454). #3519 may fix this, but I recently was reminded that Mill supports multiple parallel targets.

The issue

So I tried this command instead:

mill -j 0 -w app_lrbcol.runBackground + app_js.getScalaJs

However this doesn't do what I want. If I make a change to ScalaJS code, even if it doesn't compile, it still causes runBackground to run.

It would be better IMO if somehow watch mode could apply to each target independently, instead of what it seems to be doing, namely aggregating a single watch list of files and any of them cause the compound target to run.

[arturaz]

I'll grab this, as this is a feature I directly need.

[lihaoyi]

@arturaz go for it. I just updated the PR description with some other ideas in how to implement this

[lihaoyi]

Possible Approaches:

1. Start with runner/src/mill/runner/Watching.scala.

   • We need to make watchLoop's evaluate callback able to take an optional list of changed tasks IDs

   • watchAndWait and statWatchWait would need to return the task IDs that changed so we can pass it to evaluate. We need need to make sure we include any tasks whose methodCodeHashSignatures(methodName) changed, indicating a code change rather than an input file change

   • The evaluate callback in MillMain.scala currently runs new MillBuildBootstrap(...).evaluate(); we need to make it take the list of changed task IDs and only evaluate tasks which are downstream of the changed tasks.

   • This ability to filter a -w/--watch evaluation based on selected files or inputs will also be valuable in other contexts, e.g. selectively running tests based on git diff in CI.

2. Another way to do this would be to add an optional configuration somewhere to make evaluteGroupCache handle Commands similarly to Targets, i.e. to check the upstream inputs cache to see if they changed before running them.

   • That would probably need to change here to allow working with commands despite readWriterOpt being None, e.g. by just returning null

   • This flag can be set automatically on non-first runs of -w/--watch to satisfy this ticket, in future could be used elsewhere as desired, and would be less invasive than the original approach proposed above

3. Add an optional configuration to make EvaluatorCore skip parts of its task graph that are not downstream of union(changed_input_tasks + tasks_whose_code_sig_changed).

   • EvaluatorCore already has the task graph, sorts it in topological order, and knows where the inputs are and which tasks had their code signature changed.

   • We would (a) up-front scan all inputs for file changes and tasks for code signature changes, then (b) do a BFS downstream from those to find all tasks that could be affected, (c) do a BFS upstream from the affected tasks to find all tasks required to be run, and then (d) only bother evaluating those required tasks in topological order

[lihaoyi]

@arturaz I think approach (3) above is the most promising (least edge cases) and easiest to implement (most localized change), so I would suggest try that first