Make `out/` folder contents (more) reproducible and filesystem layout agnostic (1500USD bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 1500USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

The goal of this ticket is to make the out/ folder contents more reproducible, such that it contains the same bytes and hashes regardless of the user's filesystem layout outside of that folder. This is would allow re-using the out/ folder as a build cache between different machines that may have the checkout in different place (e.g. /Users/alice/my-repository vs /Users/charlie/my-repository), both coarse grained (e.g. by sending over a zip file) and fine grained (via the bazel remote cache protocol)

The main thing that needs to happen is that every os.Path and mill.api.PathRef that is serialized within a "known" directory needs to be normalized to a path relative to an abstract reference to that known directory. e.g.

• /Users/alice/my-repository/out/foo/bar.dest/qux should be serialized as $WORKSPACE/out/foo/bar.dest/qux
• /Users/lihaoyi/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.14/scala-library-2.13.14.jar should be serialized as $COURSIER_CACHE/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.14/scala-library-2.13.14.jar
• /Users/alice/thing-outside-repository should be serialized as $HOME/thing-outside-repository

AFAIK the necessary known roots should all be available globally (e.g. mill.api.workspace.WorkspaceRoot.workspaceRoot, os.home, sys.env("COURSIER_CACHE")). It should be easy enough to add to the serialization logic:

• mill.api.PathRef serialization https://github.com/com-lihaoyi/mill/blob/e0a2c93bfbc7bf68a71ddbc0c52afbb14e73e6f2/main/api/src/mill/api/PathRef.scala#L175-L197
• os.Path serialization https://github.com/com-lihaoyi/mill/blob/e0a2c93bfbc7bf68a71ddbc0c52afbb14e73e6f2/main/api/src/mill/api/JsonFormatters.scala#L27-L31

Apart from PathRef and Path, we will also need to deal with:

• Files in out/ which are naturally non-deterministic: mill-profile.json, mill-chrome-profile.json, mill-server/* and mill-no-server/*, etc.

• Modified times are also expected to vary. These may need to be zeroed out in the process of making zip and jar files such that they do not affect the byte contents, and ignored as part of any equivalence comparison

• Any foo.json files belonging to workers can also be expected to differ since they contain the toString of the worker, and may need to be renamed to foo.worker.json or similar to make them identifiable.

• There will also be inherent differences between files generated on different platforms (e.g. native binaries). This is fine for now, and likely unavoidable.

• There may be other files that need to be made reproducible that are not listed here

The success criteria would be a test in integration/feature/ that:

• Copies the code in example/scalalib/web/5-webapp-scalajs-shared into two separate subfolders.
  • The choice of example/scalalib/web/5-webapp-scalajs-shared is somewhat arbitrary, but should give us good coverage of a variety of Mill module and task types, exercising a wide range of code paths
• Runs ./mill runBackground && ./mill clean runBackground && ./mill jar && ./mill assembly in each folder
  • (one with a custom COURSIER_CACHE and -Duser.home passed in),
• Does a file-by-file and byte-for-byte comparison against the two outfolders with some normalization criteria (ignoring the expected-to-differ files and ignoring mtimes), to assert that the out/ folder is byte-for-byte identical

Related issues with prior discussion:

• https://github.com/com-lihaoyi/mill/issues/2101
• https://github.com/com-lihaoyi/mill/issues/2153
• https://github.com/com-lihaoyi/mill/issues/1886

[rahat2134]

Trying this issue, it looks interesting...

[lihaoyi]

@rahat2134 got for it! Feel free to ask here if you have any questions