amient / affinity

Library and a Framework for building fast, scalable, fault-tolerant Data APIs based on Akka, Avro, ZooKeeper and Kafka
Apache License 2.0
26 stars 4 forks source link

Avro: Consider Value Classes as types with logicalType of its primitive #274

Closed michal-harish closed 5 years ago

michal-harish commented 5 years ago

Currently if we have a domain like this:

case class ServiceID(id: String) extends AnyVal
case class Hostname(name: String) extends AnyVal
case class Service(id: ServiceID, host: Hostname) extends AvroRecord

Generating schema AvroRecord.inferSchema[Service] will look like this:

{
  "type":"record",
  "name":"Service",
  "namespace":"io.amient.affinity.avro",
  "fields":[
    {
      "name":"id",
      "type":{
        "type":"record",
        "name":"ServiceID",
        "fields":[ {"name":"id", "type":"string"}]
      }
    },
    {
      "name":"host",
      "type":{
        "type":"record",
        "name":"Hostname",
        "fields":[ {"name":"name", "type":"string"}]
      }
    }
  ]
}

Serialization/Deserialization of this is not working but it's not difficult to do so, however the question is whether this is the right representation of value classes. It would be nicer to represent this as a primitive whose logicalType is the value class name. This doesn't necessarily port outside java but other languages can simply take the primitive value form. It needs to be tested whether custom/dynamic logical types are actually allowed by the avro specification. If yes we could also port types currently using runtime attribute to represent custom logical types.

The above schema with logialType applied would look like this:

{
  "type":"record",
  "name":"Service",
  "namespace":"io.amient.affinity.avro",
  "fields":[
    {
      "name":"id",
      "type":"string", 
      "logicalType": "io.amient.affinity.avro.ServiceID"
    },
    {
      "name":"host",
      "type":"string",
      "logicalType": "io.amient.affinity.avro.Hostname"
    }
  ]
}