Reuse types

[fxn]

The README has examples for one type.

I have two types, priv/.../Event.avsc, and priv/.../EventType.avsc. The latter is an enum, and Event has a field of type EventType.

Encoding an event yields

[debug] reading schema `...Event` from the file /app/priv/.../Event.avsc
** (throw) {:ref_to_unknown_type, "...EventType"}

Is there a way to get that working?

[Strech]

@fxn Hey, sorry for a long pause, got a bit sick. I might know what you mean. A type lookup over different files. I think avro_turf gem even allows you to split one schema into multiple files and assemble them into one for registration in a registry.

Let me try to fix it 🔥

[Strech]

Answering question

Is there a way to get that working?

Yes, you need to add schema definition of EventType to Event schema (at least once, better to the most top place of usage)

And may I ask you to post a sample of those maybe without extra fields, to get how you try to use it?

[fxn]

Wait, perhaps I am looking for something that doesn't square with Avro.

If you embed the type definition as you do in this fixture for io.confluent.Message, but in a different type you also have something of type io.confluent.Message, you have to duplicate the type definition there?

[fxn]

Guess I was using that feature in avro_turf without even noticing :).

[Strech]

@fxn Exactly, it works magically behind the scenes in avro_turf. I was postponing this feature till the last, but now I have an additional reason to work on it 😜

[Strech]

@fxn Good news, I manage to have a working draft, but with some differences from avro_turf end-result (compiled schema is more standardized than in ruby). I hope to publish something you can try in the closest weekends ✌️

[fxn]

That's awesome ❤️.

[Strech]

@fxn Hey, it took me a while (a bit of family travelling 🌴), but here you can check not fully finished (few optimizations are missing) inter-schema refefences feature https://github.com/Strech/avrora/pull/18

The not-so-optimal part is a N-read of the reference if it's reused in different schemas (so IO on a first read) but rest should work. Can you try it and give feedback?

Only one important thing – reference should be a full name = namespace + name, i.e

{
  "namespace": "io.confluent",
  "type": "record",
  "name": "Messenger",
  "fields": [
    {
      "name": "messages",
      "type": {
        "type": "array",
        "items": "io.confluent.Message" // <--- full name reference (namespace + name)
      }
    }
  ]
}

[Strech]

Hola @fxn, a new release with inter-schema references (now complete version) was published https://github.com/Strech/avrora/releases/tag/v0.8.0

Any feedback appreciated! Happy coding ❤️

[fxn]

Sorry for not responding earlier, thanks a lot for working on this, I'll run some tests and wil write back!

[fxn]

Modulus Christmas break, I have been using 0.8.0 since it was released without any issues. Have checked some cross-references to records and enums.

Thanks for working on this ❤️.

[Strech]

@fxn Woohooo, great news 😂