search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
127 stars 1.04k forks source link

Ksql streams created using CSAS statements create topic automatically with some default schema, is there any way to override this ? #1972

Open Worm4047 opened 6 years ago

miguno commented 6 years ago

What do you mean by "with some default schema"?

You can customize settings such as number of partitions or the replication factor via the WITH clause of CSAS. See https://docs.confluent.io/current/ksql/docs/developer-guide/syntax-reference.html#create-table-reserved.

Worm4047 commented 6 years ago

Yes I tried that, but even after using WITH clause, a new schema is created and takes over the previous schema. The new schema is from ksql data source.

miguno commented 6 years ago

Sorry, I meant: What is "the schema" for you? KSQL's internal data schema? An Avro schema? Or do you mean sth else?

Worm4047 commented 6 years ago

My topic has avro schema. And one of the field is like : Id: 1 But when CSAS is used the schema is changed to this form: Id: {int,1}

TobiasHauck commented 5 years ago

That's exactly what I'm experiencing right now.

Scenario: You've got two topics that are populated by kafka REST and they've got the same schema. Then you've got a stream streaming the data from one topic to the other based on avro.

The data coming from kafka REST look like {"key": "value"}, the data coming from the stream looks like {"key": {"string":"value"}}. The reason is: KSQL transforms all fields into nullable fields which means the data structure changes.

Is there a way to configure KSQL not to treat everything as nullable, but to use exactly the schema it relies on?

vcrfxia commented 5 years ago

Related, since if KSQL can use an existing schema then we have a workaround for this issue: https://github.com/confluentinc/ksql/issues/2427