Infer schemas

[mtth]

Adding a function (or stream) to infer Avro types from values.

For example:

avro.infer([123, 3.5]); // <FloatType>
avro.infer(['hi', ['hey']]); // <UnionType ["string", {"type": "array", "items": "string"}]>

Exposing a stream interface would be helpful as well. That would open up possibilities like compressing arbitrary long collections of records into an Avro file, without requiring a schema upfront (e.g. for logs).

Possible signature:

infer(vals, [opts])

• vals {Array} Array of values to infer the type for.
• opts {Object} Options:
  • objectHook {Function} Hook called each time an unrecognized object is encountered. It can for example be used to give explicit names to generated records or infer schemas which include logical types.
  • thresholds {Object} Map of thresholds used to determine when to switch from records to maps, enums to strings.
  • noNullDefault {Boolean} By default missing record fields will be coerced to null default values. This option suppresses this behavior. Setting this option might lead to infeasible inference.

[dceejay]

Would be great, but how would decode then work ? Or would you just unpack without a schema to check against ?

[mtth]

@dceejay - It depends on the use-case. For example:

• If writing to a file, the schema could be directly stored with the data. We could even reuse Avro's container file format in that case.
• If something more dynamic is required, schemas could be assigned an ID and stored in a registry. Prefixing each encoded value with its ID would let us later decode them. This approach is similar to Confluent's schema registry often used with Kafka.

Since many people aren't familiar with Avro's schema notation, having this schema-inferring ability would also make it easier to get started. You would just put in a few representative values and then use the generated schema as you wish. For complex structured data where the generated schema might not be optimal, it still provides a starting point which you can edit, which is much easier than starting from scratch.

[dceejay]

ab-so-lutely ! :+1: Ah - so this would be to build the schema... it's not (yet...) auto encode to binary format... Though I guess you could pipeline it.

hmm - I wonder if there would be a way to "fingerprint" each schema / message to auto create an ID, such that subsequent messages of same pattern can reuse that schema (and not incur generation overhead per message) - and get labelled with ID.

[mtth]

so this would be to build the schema... it's not (yet...) auto encode to binary format... Though I guess you could pipeline it.

Exactly.

I wonder if there would be a way to "fingerprint" each schema / message to auto create an ID, such that subsequent messages of same pattern can reuse that schema

This is definitely possible. In fact the fingerprint part is already implemented: type.getFingerprint.

[dceejay]

nice. Can't wait to try it :-)

I guess there is no obvious way to do an enum - but as you said it would be a great way to get started.

[ms440]

That would be great!

[mtth]

Small update: the public API for this isn't quite ready yet, but the repo now includes a script to infer a schema from a given value, in case you'd like to try it out. Sample commands below.

$ ./etc/scripts/infer '[1, 2, 3]' # An array of integers.
{"type":"array","items":"int"}

$ ./etc/scripts/infer '[1, 2, 3, 4.5]' # Adding a float.
{"type":"array","items":"float"}

$ ./etc/scripts/infer '{"id": 48, "on": true}' # A sample record.
{"name":"_0","type":"record","fields":[{"name":"id","type":"int"},{"name":"on","type":"boolean"}]}

[mtth]

Released as part of 4.1.0!

Helpful links:

• Example in the README
• Short documentation section
• Standalone type inference script

In particular the last script will let you infer the type from any file containing JSON values:

$ cat example.jsonl
{"id": 12, "title": "Hello"}
{"id": 3}
{"id": 4, "title": "Hey"}

$ ./etc/scripts/infer <example.jsonl
{
  "type": "record",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "title", "type": ["null", "string"], "default": null}
  ]
}

[dceejay]

excellent - thanks.