Cannot create classes from schema with references to other schemas

[ticofab]

This issue might be related to the underlying Apache library rather than sbt-avrohugger (or avrohugger), I'm not sure.

Problem

I can't generate case classes for schemas that contains arrays of referenced schemas.

Reproduction steps

Using Scala 3.3.1 and sbt-avrohugger 2.8.2, and the following objects:

Package.avsc

{
  "name": "Package",
  "namespace": "test",
  "type": "record",
  "fields": [
    {
      "name": "trackingCode",
      "type": "string"
    }
  ]
}

Two other schemas reference this object:

{
  "name": "ShipmentDeliveryHasFailed",
  "namespace": "test",
  "type": "record",
  "fields": [
    {
      "name": "packages",
      "type": {
        "type": "array",
        "items": {
          "name": "package",
          "type": "test.Package"
        }
      }
    }
  ]
}

and

{
  "name": "ShipmentHasBeenDelivered",
  "namespace": "test",
  "type": "record",
  "fields": [
    {
      "name": "packages",
      "type": {
        "type": "array",
        "items": {
          "name": "package",
          "type": "test.Package"
        }
      }
    }
  ]
}

The generation of case classes fails with

[error] org.apache.avro.SchemaParseException: Type not supported: test.Package
[error]     at org.apache.avro.Schema.parse(Schema.java:1797)
[error]     at org.apache.avro.Schema.parse(Schema.java:1779)
[error]     at org.apache.avro.Schema.parse(Schema.java:1736)
[error]     at org.apache.avro.Schema$Parser.parse(Schema.java:1471)
[error]     at org.apache.avro.Schema$Parser.parse(Schema.java:1433)
[error]     at avrohugger.input.parsers.FileInputParser.$anonfun$getSchemaOrProtocols$3(FileInputParser.scala:68)
[error]     at scala.util.Try$.apply(Try.scala:213)
[error]     at avrohugger.input.parsers.FileInputParser.tryParse$1(FileInputParser.scala:68)
[error]     at avrohugger.input.parsers.FileInputParser.getSchemaOrProtocols(FileInputParser.scala:86)
[error]     at avrohugger.generators.FileGenerator$.fileToFile(FileGenerator.scala:88)
[error]     at avrohugger.Generator.fileToFile(Generator.scala:80)
[error]     at sbtavrohugger.FileWriter$.$anonfun$generateCaseClasses$4(FileWriter.scala:26)
[error]     at sbtavrohugger.FileWriter$.$anonfun$generateCaseClasses$4$adapted(FileWriter.scala:24)
[error]     at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[error]     at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[error]     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[error]     at sbtavrohugger.FileWriter$.generateCaseClasses(FileWriter.scala:24)
[error]     at sbtavrohugger.SbtAvrohugger$.$anonfun$avroSettings$8(SbtAvrohugger.scala:82)
[error]     at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:81)
[error]     at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:154)
[error]     at sbt.util.Difference.apply(Tracked.scala:415)
[error]     at sbt.util.Difference.apply(Tracked.scala:395)
[error]     at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:150)
[error]     at sbt.util.Difference.apply(Tracked.scala:415)
[error]     at sbt.util.Difference.apply(Tracked.scala:390)
[error]     at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:149)
[error]     at sbtavrohugger.SbtAvrohugger$.$anonfun$avroSettings$7(SbtAvrohugger.scala:84)
[error]     at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]     at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error]     at sbt.Execute.work(Execute.scala:292)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error]     at java.base/java.lang.Thread.run(Thread.java:1583)
[error] (Compile / avroScalaGenerate) org.apache.avro.SchemaParseException: Type not supported: test.Package

Notes

The generation works if the referenced object Package is a single object instead of an array, like this:

{
  "name": "ShipmentHasBeenDelivered",
  "namespace": "test",
  "type": "record",
  "fields": [
    {
      "name": "packages",
      "type": "test.Package"
    }
  ]
}

So the problem seems to lie in the generation of array of referenced objects. I am basically trying to achieve what this SO answer achieves. Tried with previous version of sbt-avrohugger and met the same problem.

Thank you for the work, the attention and the support.

[julianpeeters]

Hi Fabio, thanks for the report

That org.apache.avro.SchemaParseException error is indeed coming from avro, not avrohugger. And it's trying to say that there's some nonsense in the schema.

Hard to tell, by eye. Luckily this test popped up, using GitHub search: https://github.com/julianpeeters/sbt-avrohugger/blob/122b9d5faea05542f1b4280a51d0e8c5436e7e55/src/test/resources/dependencies/ArrayRef.avsc#L13

So you may like to try redefining your schema, to match the format in that example

"fields" : [
    {
      "name": "ancestors",
      "type": {
        "type": "array", 
        "items": "common.Z"
      }
    }
  ]

[ticofab]

Wow @julianpeeters, thank you for the pointer. I revised the schema definition and it worked. I am very thankful for your time and support 🙏