ybyzek
commented
3 years ago If we wanted to tie this in with streaming processing, then for ksqlDB:
ksqlDB does not have much support for topics with multiple schema types right now: https://github.com/confluentinc/ksql/issues/1267.
Potential workaround from @mikebin
That said, I think if you have a union/oneOf schema defined as in the blog post, ksqlDB will infer a “superset” schema containing all the types in the union. For example, with a schema like this:
{
"fields": [
{
"name": "oneof_type",
"type": [
{
"fields": [
{
"name": "fname",
"type": "string"
},
{
"name": "lname",
"type": "string"
}
],
"name": "Customer",
"type": "record"
},
{
"fields": [
{
"name": "city",
"type": "string"
},
{
"name": "state",
"type": "string"
}
],
"name": "Address",
"type": "record"
}
]
}
],
"name": "AllTypes",
"namespace": "io.confluent.examples.avro",
"type": "record"
}ksqlDB would infer a schema that looks like this:
Field | Type
-----------------------------------------------------------------------------------------------------------------------------
ONEOF_TYPE | STRUCT<CUSTOMER STRUCT<FNAME VARCHAR(STRING), LNAME VARCHAR(STRING)>, ADDRESS STRUCT<CITY VARCHAR(STRING), STATE VARCHAR(STRING)>>So you could possibly write queries with CASE expressions to check which of the types are populated (i.e., non-null) for each record. I haven’t gone through an actual implementation of this, so perhaps someone else on the channel will have better insight and suggestions.
pbarsotti-glovo
bbejeck
Schema Registry 5.5.0 supports having multiple event types per topic. This tutorial would cover what's needed to enable this functionality and when to use it. This feature builds off schema references, so #623 should be done first.