search

julianpeeters / sbt-avrohugger

sbt plugin for generating Scala sources for Apache Avro schemas and protocols.
Apache License 2.0
133 stars 50 forks source link

java.lang.RuntimeException: Problem using the file: java.nio.file.FileSystemException: path_to_file.scala: Invalid argument #94

Closed mcenkar closed 1 year ago

mcenkar commented 1 year ago

Hi,

when upgrading sbt-avrohugger from 2.2.1 to 2.3.0 I'm faced with following error:

[error] java.lang.RuntimeException: Problem using the file: java.nio.file.FileSystemException: /.../workspace/website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/RegularTradingHourEvent.scala: Invalid argument
[error]     at scala.sys.package$.error(package.scala:30)
[error]     at avrohugger.format.abstractions.SourceFormat.writeToFile(SourceFormat.scala:231)
[error]     at avrohugger.format.abstractions.SourceFormat.writeToFile$(SourceFormat.scala:218)
[error]     at avrohugger.format.SpecificRecord$.writeToFile(SpecificRecord.scala:21)
[error]     at avrohugger.format.SpecificRecord$.$anonfun$compile$1(SpecificRecord.scala:222)
[error]     at avrohugger.format.SpecificRecord$.$anonfun$compile$1$adapted(SpecificRecord.scala:222)
[error]     at scala.collection.immutable.List.foreach(List.scala:431)
[error]     at avrohugger.format.SpecificRecord$.compile(SpecificRecord.scala:222)
[error]     at avrohugger.generators.FileGenerator$.protocolToFile(FileGenerator.scala:50)
[error]     at avrohugger.generators.FileGenerator$.$anonfun$fileToFile$1(FileGenerator.scala:94)
[error]     at avrohugger.generators.FileGenerator$.$anonfun$fileToFile$1$adapted(FileGenerator.scala:89)
[error]     at scala.collection.immutable.List.foreach(List.scala:431)
[error] (website-server-api / Compile / avroScalaGenerateSpecific) Problem using the file: java.nio.file.FileSystemException: /.../workspace/website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/RegularTradingHourEvent.scala: Invalid argument
[error] Total time: 27 s, completed 3 Mar 2023, 08:22:40

seems like running an extra compile does solve the problem. However it's a bit difficult on CI + it was working fine on 2.2.1.

Did anything change in how sbt-avrohugger needs to be plugged in? My build is probably a bit complicated because it's compiling avdl files extracted from jar by another sbt task, so possible there's some error on my part.

// edit: actually it seems to be a bit nondeterministic as sometimes builds pass just fine

julianpeeters commented 1 year ago

Hi @mcenkar, sorry you're experiencing this, and thanks for reporting.

I don't have very helpful insight, unfortunately:

  1. The changes from 2.2.1 to 2.3.0 were very minimal, but possibly the nondeterminism was made visible by a the bump to sbt 1.8.2
  2. sbt-avrohugger has long suffered from a known non-determinism, and, while your symptoms are different, the concurrency workaround may be a long shot.
  3. If your issue is another manifestation of (2), then the current plan is to use a Ref to solve the issue, but that's a little further down the roadmap
  4. I could be way off-base with the above, as I haven't been able to reproduce these issues myself
mcenkar commented 1 year ago

Actually had a longer look and it's not 2.3.0, I had it even on 2.0.0 - only noticed now, or maybe was just unlucky to get it couple times in a row.

Maybe narrowed down nondeterminism to:

[info] Compiling Avro IDL /.../website-protobuf/server-api/src/main/generated/avro/ListedOptionIdentifiersEvents.avdl to /.../website-protobuf/server-api/target/src_managed/main/compiled_avro
[info] Compiling 2 protobuf files to /.../website-protobuf/server-api/target/src_managed/main
[error] java.lang.RuntimeException: Problem using the file: java.nio.file.NoSuchFileException: /.../website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/ListedOptionIdentifiersEvent.scala
[error]     at scala.sys.package$.error(package.scala:30)
[error]     at avrohugger.format.abstractions.SourceFormat.writeToFile(SourceFormat.scala:231)

and sometimes it manages to compile 2 files (out of 6 total):

[info] Compiling Avro IDL /.../website-protobuf/server-api/src/main/generated/avro/ListedOptionIdentifiersEvents.avdl to /.../website-protobuf/server-api/target/src_managed/main/compiled_avro
[info] Compiling Avro IDL /.../website-protobuf/server-api/src/main/generated/avro/ListedOptionIdentifiersEventV2.avdl to /.../website-protobuf/server-api/target/src_managed/main/compiled_avro
[info] Compiling 2 protobuf files to /.../website-protobuf/server-api/target/src_managed/main
[error] java.lang.RuntimeException: Problem using the file: java.nio.file.NoSuchFileException: /.../website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/ListedOptionIdentifiersEventV2.scala
[error]     at scala.sys.package$.error(package.scala:30)

we're pretty much here:

[error] java.lang.RuntimeException: Problem using the file: java.nio.file.NoSuchFileException: /.../website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/ListedOptionIdentifiersEventV2.scala
[error]     at scala.sys.package$.error(package.scala:30)
[error]     at avrohugger.format.abstractions.SourceFormat.writeToFile(SourceFormat.scala:231)

https://github.com/julianpeeters/avrohugger/blob/v1.3.0/avrohugger-core/src/main/scala/format/abstractions/SourceFormat.scala#L231

which doesn't quite make much sense to me, maybe to you:

      Files.deleteIfExists(path)
      Files.write(path, contents, StandardOpenOption.CREATE)

first one shouldn't throw an error, second one shouldn't throw NoSuchFileException because it should just create the file.

// edit Actually second would throw this exception if entire folder didn't exist. So it's supposed to write to /a/b/c.txt but there's only /a/ => it will throw an error.

@julianpeeters maybe there's too much deleting somewhere?

full stacktrace for reference:

[error]     at scala.sys.package$.error(package.scala:30)
[error]     at avrohugger.format.abstractions.SourceFormat.writeToFile(SourceFormat.scala:231)
[error]     at avrohugger.format.abstractions.SourceFormat.writeToFile$(SourceFormat.scala:218)
[error]     at avrohugger.format.SpecificRecord$.writeToFile(SpecificRecord.scala:21)
[error]     at avrohugger.format.SpecificRecord$.$anonfun$compile$1(SpecificRecord.scala:222)
[error]     at avrohugger.format.SpecificRecord$.$anonfun$compile$1$adapted(SpecificRecord.scala:222)
[error]     at scala.collection.immutable.List.foreach(List.scala:431)
[error]     at avrohugger.format.SpecificRecord$.compile(SpecificRecord.scala:222)
[error]     at avrohugger.generators.FileGenerator$.protocolToFile(FileGenerator.scala:50)
[error]     at avrohugger.generators.FileGenerator$.$anonfun$fileToFile$1(FileGenerator.scala:94)
[error]     at avrohugger.generators.FileGenerator$.$anonfun$fileToFile$1$adapted(FileGenerator.scala:89)
[error]     at scala.collection.immutable.List.foreach(List.scala:431)
[error]     at avrohugger.generators.FileGenerator$.fileToFile(FileGenerator.scala:89)
[error]     at avrohugger.Generator.fileToFile(Generator.scala:80)
[error]     at sbtavrohugger.FileWriter$.$anonfun$generateCaseClasses$8(FileWriter.scala:37)
[error]     at sbtavrohugger.FileWriter$.$anonfun$generateCaseClasses$8$adapted(FileWriter.scala:34)
[error]     at scala.collection.immutable.List.foreach(List.scala:431)
[error]     at sbtavrohugger.FileWriter$.generateCaseClasses(FileWriter.scala:34)
[error]     at sbtavrohugger.SbtAvrohugger$.$anonfun$specificAvroSettings$8(SbtAvrohugger.scala:160)
[error]     at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:80)
[error]     at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:153)
[error]     at sbt.util.Difference.apply(Tracked.scala:414)
[error]     at sbt.util.Difference.apply(Tracked.scala:394)
[error]     at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:149)
[error]     at sbt.util.Difference.apply(Tracked.scala:414)
[error]     at sbt.util.Difference.apply(Tracked.scala:389)
[error]     at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:148)
[error]     at sbtavrohugger.SbtAvrohugger$.$anonfun$specificAvroSettings$7(SbtAvrohugger.scala:161)
[error]     at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]     at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error]     at sbt.Execute.work(Execute.scala:291)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error]     at java.base/java.lang.Thread.run(Thread.java:1589)
mcenkar commented 1 year ago

@julianpeeters maybe there's too much deleting somewhere?

That was actually pretty good guess, if I create all folders on-the-way to actual one I don't get any more errors.

https://github.com/julianpeeters/avrohugger/pull/165

Not sure if there might be another bug - because why it's nondeterministic - can something delete folder when files were already generated - but it does fix problem for me.

julianpeeters commented 1 year ago

@mcenkar thanks a ton for looking into this. The fix has been released as avrohugger version 1.3.1 and sbt-avrohugger version 2.3.1