Closed rodesai closed 1 year ago
@bvarghese1 , @spena mentioned this is similar to a fix he's working on for https://github.com/confluentinc/ksql/issues/9169 . Maybe you wan to wait before starting this work ?
Will sync with @spena regarding this. Thank you for the update @suhas-satish
Describe the bug ksqlDB cannot execute a JOIN when one of the sources uses a schema referenced by ID. This is because of the interaction of these 2 behaviors:
When a join requires a repartition, ksqlDB will try to use the physical schema of the source being repartitioned to write to the repartition topic, if the source has a schema ID
For joins, ksqlDB first does a projection to a new schema that prefixes each column with the source name. So, for example if I do:
Then, the repartition schema has columns like
SOURCE_A STRING, SOURCE_B STRING, SOURCE2_A STRING, SOURCE2_B STRING
The problem here is that the source schema referenced by ID does not have fields that are prefixed with the source name. So when ksqlDB goes to serialize the records for writing to the repartition topic, the serializer complains that there is no field in the schema that matches the field name in the record:
To Reproduce Here's a QTT:
Expected behavior The join should execute correctly
Additional context I think the right approach here is not to forward the schema id along to the repartition serdes. They should just generate their own schema to use for the repartition topic.