AvscFileSorter not taking nullable array into account

[sophiewachs]

Hi,

I just run into an issue adding one new nested level in my avsc files. I used the 0.9.1 version. It seems that this kind of type

"type": [
        "null",
        {
          "type": "array",
          "items": "model.IdAndLabel"
        }
      ]

does not allow model.IdAndLabel to be taken into account by the AVSCFileSorter.

If the null is removed, the files are treated in the correct order

"type":
        {
          "type": "array",
          "items": "model.IdAndLabel"
        }

I had 3 levels of nested object like the following. The father.avsc is treated before idAndLabel.avsc resulting in the following error : org.apache.avro.SchemaParseException: Undefined name: "model.IdAndLabel"

idAndLabel.avsc

{
  "type": "record",
  "name": "IdAndLabel",
  "namespace": "model",
  "fields": [
    {
      "name": "id",
      "type": {
        "type": "string"
      }
    },
    {
      "name": "label",
      "type": {
        "type": "string"
      }
    }
  ]
}

father.avsc

{
  "type": "record",
  "name": "Father",
  "namespace": "model",
  "fields": [
    {
      "name": "idsAndLabels",
      "type": [
        "null",
        {
          "type": "array",
          "items": "model.IdAndLabel"
        }
      ]
    }
  ]
}

grandFather.avsc

{
  "type": "record",
  "name": "GrandFather",
  "namespace": "model",
  "fields": [
    {
      "name": "father",
      "type": "model.Father"
    }
  ]
}

Thanks !

[julianpeeters]

Hi Sophie, you should be set to go with sbt-avrohugger version 0.9.2, syncing up with maven central shortly.

Sorry for the delay, and thanks very much for the report and example schemas. Please let me know if this doesn't fix your issue, or if you have any other ideas about improvements.

Cheers, Julian

[sophiewachs]

Hi Julian,

Thanks for the fix. I just tested the version 0.9.2 and it worked fine for my issue.

However, it seems to remove the possibility to define the array "items" as a nested avro; which was the workaround I used for my issue

here is the avsc I used

{
    "type": "record",
    "name": "Father",
    "namespace": "model",
    "fields": [{
        "name": "idsAndLabels",
        "type": [
            "null", {
                "type": "array",
                "items": {
                    "type": "record",
                    "name": "IdAndLabel",
                    "namespace": "model",
                    "fields": [{
                        "name": "id",
                        "type": {
                            "type": "string"
                        }
                    }, {
                        "name": "label",
                        "type": {
                            "type": "string"
                        }
                    }]
                }
            }
        ]
    }]
}

And the error I get with the 0.9.2 version

spray.json.DeserializationException: Expected String as JsString, but got {"type":"record","name":"IdAndLabel","namespace":"model","fields":[{"name":"id","type":{"type":"string"}},{"name":"label","type":{"type":"string"}}]}
    at spray.json.package$.deserializationError(package.scala:23)
    at spray.json.BasicFormats$StringJsonFormat$.read(BasicFormats.scala:126)
    at spray.json.BasicFormats$StringJsonFormat$.read(BasicFormats.scala:119)
    at spray.json.JsValue.convertTo(JsValue.scala:31)
    at sbtavrohugger.filesorter.ReferredTypeFinder$$anonfun$2$$anonfun$apply$3.sbtavrohugger$filesorter$ReferredTypeFinder$$anonfun$$anonfun$$matchComplexType$1(ReferredTypeFinder.scala:37)
    at sbtavrohugger.filesorter.ReferredTypeFinder$$anonfun$2$$anonfun$apply$3$$anonfun$3.apply(ReferredTypeFinder.scala:53)
    at sbtavrohugger.filesorter.ReferredTypeFinder$$anonfun$2$$anonfun$apply$3$$anonfun$3.apply(ReferredTypeFinder.scala:48)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)

[julianpeeters]

Oh no! Thanks again for the heads up. Support for nested schema definitions should now be restored in version 0.9.3