Support SCHEMA_ID in WITH clause

[agavra]

Is your feature request related to a problem? Please describe.

If I have a schema that is in schema registry but is not under the correct subject name, I cannot specify my KSQL stream to use that topic with AVRO format.

Describe the solution you'd like We should support specifying KEY_SCHEMA_ID and VALUE_SCHEMA_ID in the WITH clause of create statements to support this use case.

Note that the KEY_SCHEMA_ID and VALUE_SCHEMA_ID properties actually already exist in the codebase today, but they are not fully functional: they may be used to specify schema IDs but those IDs must still be registered under the appropriate subject names or else the statement will be rejected. Because the functionality of these properties is only partially complete as a result, the configs are not documented / officially supported at this time.

Describe alternatives you've considered

• Specifying Subject Name
• Changing the "naming strategy". This is the way that connect does it, but I think that's because they also use that to generate schemas. ID is clearer.

[AustinHunt]

Our team finds this useful. We encode several different message types into a single topic. Being able to select which schema the query uses would be useful.

[big-andy-coates]

Allowing users to specify the schema id in the with clause is one way solution to stop ksql overwriting an existing schema.

There are many requests for something similar. We should find the best solution for this.

[Petazetas754]

Allowing users to specify the schema id in the with clause is one way solution to stop ksql overwriting an existing schema.

There are many requests for something similar. We should find the best solution for this.

This would be very useful for our organisation too. We are currently having issues with KSQL overwriting an existing schema when creating a TABLE over a topic which is being populated via the Spool Dir connector.

If we CREATE the table in BACKWARD compatibility mode this is ok, it just creates a new version of the schema, but in FULL compatibility it fails stating it cannot write the schema in the registry.

Ideally we would like for the KSQL TABLE to reuse the existing schema as opposed to try to overwrite it or create a new version

[big-andy-coates]

Another option would be to allow users to tell ksqlDB to use the latest registered version of the schema. This won't solve use-cases that want to use schemas registered under a different subject, but will allow users to avoid ksql overwriting existing schemas, i.e. both this and adding an explicit SCHEMA_ID may provide greater flexibility.

[mikebin]

Is the intent for this ticket to only address the source schema, or also the output schema? There are some other tickets linked here which are more associated with output schema:

https://github.com/confluentinc/ksql/pull/2679 https://github.com/confluentinc/ksql/issues/5256

[agavra]

another ticket related: https://github.com/confluentinc/ksql/issues/4517

[ThomasFerro]

Hi team,

We would also like this feature since we lost important information in the generated schema (every field is nullable and without the original documentation).

I cannot see how the described alternatives can solve the issue, did specifying a name prevent ksql from creating new schemas ? If so, could you please provide us a link to the related documentation ? We do not seems to find related information on https://docs.ksqldb.io/ but we have just started using the tool and his documentation.

Thanks !

[vcrfxia]

Hey @lihaosky this work is complete now as of the upcoming ksqlDB 0.24 release, right? I'm going to close this -- please reopen if it's not complete. Thanks!

[PBholewasi]

Hey @agavra after using VALUE_SCHMA_ID I'm getting serialization error. It says the schema's are incompatible.

[lihaosky]

Hi @PBholewasi , please file a new issue with more details. I will look into it. Thanks!