Parallel elm make calls corrupt elm-stuff/0.19.1/i.dat

[omnibs]

Quick Summary: Elm make running in parallel corrupts elm-stuff/0.19.1/i.dat

• Elm: 0.19.1
• Browser: N/A
• Operating System: Ubuntu 16.04

Running elm-make in parallel to compile different entry points every now and then errors out with the message below:

Compiling ...+-------------------------------------------------------------------------------
|  Corrupt File: /build/source/myproject/elm-stuff/0.19.1/i.dat
|   Byte Offset: 102336
|       Message: not enough bytes
|
| Please report this to https://github.com/elm/compiler/issues
| Trying to continue anyway.
+-------------------------------------------------------------------------------

-- CORRUPT CACHE ---------------------------------------------------------------

It looks like some of the information cached in elm-stuff/ has been corrupted.

Try deleting your elm-stuff/ directory to get unstuck.

Note: This almost certainly means that a 3rd party tool (or editor plugin) is
causing problems your the elm-stuff/ directory. Try disabling 3rd party tools
one by one until you figure out which it is!

Additional Details

This has been mentioned in https://github.com/elm/compiler/issues/1830#issuecomment-727259777, we found it by searching for the error message.

We were facing the same problem when using esbuild's elm plugin. I opened an issue on the plugin's side here: https://github.com/phenax/esbuild-plugin-elm/issues/2

There might be other spots https://github.com/elm/compiler/commit/054fa04074fd819958affd04a526e10f69aa5197 didn't cover that might be worth locking on.

Work-around

Don't let elm run in parallel.

It might be hard to track who's launching elm in parallel. It was for us. To track down all places we wrapped the elm executable in a shell script that blew every elm up when launching more than one instance of elm in parallel. This way we also caught a failures in the already-running elm callsite once a parallel launch was attempted.

[github-actions[bot]]

Thanks for reporting this! To set expectations:

• Issues are reviewed in batches, so it can take some time to get a response.
• Ask questions in a community forum. You will get an answer quicker that way!
• If you experience something similar, open a new issue. We like duplicates.

Finally, please be patient with the core team. They are trying their best with limited resources.

[omnibs]

Not sure if it's relevant, but we were also observing o.dat get corrupted.

[jjant]

@evancz Any plans to make the compiler able to run in parallel?

[phenax]

Any updates on this?

[lydell]

I have one potential update. In my experience, Elm has a file based lock so that parallel elm make actually run in sequence. However, there is no lock for the “dependencies” part of the compilation – that’s where I’ve seen failures. So what I did in my case was to first run elm make --output /dev/null Dummy.elm to force the dependencies stuff to be done, then I started my parallel elm make processes.

[omnibs]

We tried serial dependency fetching -> parallel compilation, and can report it didn't work for us. We still ran into the same issue described in the OP and had to quickly revert.

I'm not contesting @lydell's report tho. The trick might indeed reduce the likelihood of corruption, but that reduction wasn't enough to prevent corruption from happening frequently in a codebase with so many Elm apps to be compiled in parallel as NoRedInk's codebase.