search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
83 stars 1.04k forks source link

Plug-in extension point for user-supplied FORMATs (custom, pluggable SerDes) #1057

Open blueedgenick opened 6 years ago

blueedgenick commented 6 years ago

There are many requests already for various additional serdes (the thing exposed in a CREATE TABLE or CREATE STREAM statement as FORMAT='xxx'), with the most popular to-date being protobuf.

I've also heard requests for thrift, xml, and json-p among others. There's also a long-tail of less often used message formats, some of which may even be specific to a particular organization - e.g. a binary-encoding to interface with some in-house mainframe system.

This request is to provide a documented interface and plug-in extension point (similar maybe to how Connect can load SMTs and their dependencies, from designated classpath, into separate classloaders ?) to allow a user to supply their own SerDe class for use in KSQL. Imagine something like CREATE STREAM foo AS SELECT ... WITH (FORMAT='com.whatever.my-personal-binary-decoder');

sean-rossignol commented 1 year ago

Bumping this. We are looking for low code solutions to satisfy some future kafka streams needs but we have a custom serializer used in a lot of places internally so ksql is not really an option for us unless custom serializers are supported.