Weird, non-deterministic `IO error while decoding <directory path> with UTF-8` compilation error

[guizmaii]

steps

mkdir bug_decoding
cd bug_decoding
git clone git@github.com:guizmaii/caliban.git
cd caliban
git checkout generate_from_code_5
sbt clean +publishM2
cd ..
git clone git@github.com:guizmaii/poc_compile_time_caliban_client_generation.git
cd poc_compile_time_caliban_client_generation
sbt clean compile # If this compilation doesn't fail, retry it. It'll surely fail at the second or third try.

problem

When compiling my sbt project, I have non-deterministic error:

On Mac/Linux:

...
[info] compiling 23 Scala sources to /Users/jules/conduktor/workspace/poc_compile_time_caliban_client_generation/modules/caliban-clients/target/scala-2.12/classes ...
[error] IO error while decoding /Users/jules/conduktor/workspace/poc_compile_time_caliban_client_generation/modules/caliban-clients/src/main/scala/poc/caliban/client/generated/posts/splitted with UTF-8: /Users/jules/conduktor/workspace/poc_compile_time_caliban_client_generation/modules/caliban-clients/src/main/scala/poc/caliban/client/generated/posts/splitted (Is a directory)
[error] Please try specifying another one using the -encoding option
[error] one error found
[error] (caliban-clients / Compile / compileIncremental) Compilation failed
...

On Windows:

...
[info] compiling 23 Scala sources to C:\Users\IEUser\Desktop\workspace\poc_compile_time_caliban_client_generation\modules\caliban-clients\target\scala-2.12\classes ...
[error] IO error while decoding C:\Users\IEUser\Desktop\workspace\poc_compile_time_caliban_client_generation\modules\caliban-clients\src\main\scala\poc\caliban\client\generated\posts\splitted with UTF-8: C:\Users\IEUser\Desktop\workspace\poc_compile_time_caliban_client_generation\modules\caliban-clients\src\main\scala\poc\caliban\client\generated\posts\splitted (Access is denied)
[error] Please try specifying another one using the -encoding option
[error] one error found
[error] (caliban-clients / Compile / compileIncremental) Compilation failed
...

notes

The modules\caliban-clients\src\main\scala\poc\caliban\client\generated\posts\splitted directory is generated by some code that I call in the plugin I'm working on, see: https://github.com/ghostdogpr/caliban/pull/1037

But the error happen after we finished generating all files and happens in a non-deterministic way 🤔

It could be because the filesystem buffer is not yet written to the disk (or something like this) so when sbt tries to read the generated directory, it's not yet present on the disk, maybe. WDYT? 🤔

[guizmaii]

I may have fixed the problem with this commit: https://github.com/ghostdogpr/caliban/pull/1037/commits/64ddaeb9cf9321ba416805f0918977182f37c59d

In this commit, I change two things:

1. The blocking operations made on the filesystem were wrapped in an IO Monad but not one dedicated to blocking operations. I fixed that by wrapping these operations in ZIO effectBlocking.
2. I used an "atomic write" strategy which consists of writing in a tmp file and atomically copy this tmp file to the desired location.

The future will tell us if these changes are enough. For now, I'm not able to reproduce the error anymore neither on my local Mac nor in my VM running Windows 7.

Should sbt recover and try to fsync the file (and/or its parent) before failing to make it more robust, maybe? For reference, see:

• https://twitter.com/QueueQueueHack/status/1436297710156402696?s=20
• https://twitter.com/QueueQueueHack/status/1436297004615757825?s=20

[guizmaii]

The commit in my previous message doesn't fix the issue. 😕

[eed3si9n]

Thanks for the report! last caliban-clients / Compile / compileIncremental might give you a stack trace, which might give us more information.

[guizmaii]

Exactly the information I needed I think @eed3si9n! Thank you so much! 🙇‍♂️

I'll give a try later when I have some time.

[guizmaii]

Bug found.

Obviously, it was my fault, not Scala nor sbt's fault:

I was effectively asking to Scala to compile a directory:

[warn] ctCalibanClient - res: Vector(/home/runner/work/poc_compile_time_caliban_client_generation/poc_compile_time_caliban_client_generation/modules/caliban-clients/src/main/scala/poc/caliban/client/generated/posts/splitted, /home/runner/work/poc_compile_time_caliban_client_generation/poc_compile_time_caliban_client_generation/modules/caliban-clients/src/main/scala/poc/caliban/client/generated/posts/CalibanClient.scala)

The fix: https://github.com/ghostdogpr/caliban/pull/1037/commits/1ca081b2964abefedb1f263b159ae1a751f9f911

Thanks for your help! 🙂 Sorry to have wasted your time. 😟 Jules

[SethTisue]

the error message could be better, though!