Need better error messaging for "Failed to serialize Avro data from topic ..."

[ybyzek]

Is your feature request related to a problem? Please describe.

Trying to do an INSERT INTO on an Avro formatted stream, three people spent a long time troubleshooting the following error message:

Failed to insert values into 'AIRCRAFT'. Could not serialize value: [ xxx ]. Error serializing message to topic: xxx. Failed to serialize Avro data from topic xxx :

We went down a lot of paths to figure out what was wrong, and it turns out the issue was (as diagnosed by @jzaralim , thank you!)

Whats happening is, the name of the registered schema is io.confluent.demo.aircraft.avro.pojo.AircraftState , but when ksqlDB does an insert, it tries to register a schema. And the schema will always be named KsqlDataSourceSchema

This is because the stream had already some data in it produced by a Java app, and then ksqlDB was doing INSERT INTO.

Describe the solution you'd like Given the amount of time spent troubleshooting, it would be an enhancement to provide better messaging that indicated what the problem was (schema name conflict) and what the solution was: adding KEY_FORMAT='KAFKA', VALUE_SCHEMA_ID=10001 to the CREATE STREAM command).

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

[spena]

Reopene issue 'cause the fix only works for Protobuf

[suhas-satish]

@aliehsaeedii , can you check if this is a duplicate of another P0 that you're already working on and link it here ?

[stebl]

Reopene issue 'cause the fix only works for Protobuf

Is this still true? I found this issue while debugging what looks to be a similar root cause.

https://github.com/confluentinc/ksql/pull/9637 looks like it only addressed the error message.