Automatic dynamic type conversion

[kishaningithub]

Background

We have a very specific use case where the data is in a CSV file and there is a separate file which stores avro schema. We will have to convert the csv data into OCF format and send it out.

The problem we face is the CSV data is array of strings, and because of this the BinaryFromNative function returns type errors like

value does not match its schema: long: expected: Go numeric; received: string

Schema

{
  "type": "record",
  "name": "example",
  "fields": [
    {
      "name": "customer",
      "type": "long",
      "default": 0
    }
  ]
}

Data

map[string]interface{}{
  "customer":  "12323423",
}

Feature request

1. Can we auto parse string to a type required by the avro schema at run time?
2. Also if a data is not present(empty string) default value must be populated

[kishaningithub]

@karrick I would understand if this library does not want to perform type fudging / dynamic conversions

In which case can this library parse the schema and give out a field representation so that another library can write the dynamic type conversions?

For example

https://github.com/allegro/json-avro-converter/blob/a50090bc25a1c234f9be07d089640a43b3c0e56c/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonGenericRecordReader.java#L94

This implements dynamic type conversion to translate json to avro

The base for this is that org.apache.avro:avro library provided a Schema.Field abstraction

The most powerful part of this abstraction was the simplicity in writing code for union types type

1. https://github.com/allegro/json-avro-converter/blob/a50090bc25a1c234f9be07d089640a43b3c0e56c/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonGenericRecordReader.java#L97
2. https://github.com/allegro/json-avro-converter/blob/a50090bc25a1c234f9be07d089640a43b3c0e56c/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonGenericRecordReader.java#L148