search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
44 stars 1.04k forks source link

Topic configuration while creating a stream/table #5148

Closed stefanfrehse closed 1 year ago

stefanfrehse commented 4 years ago

Is your feature request related to a problem? Please describe. For statements as CSAS and CTAS topics are usually automatically created. Currently, the user has no direct way to change the default topic configuration directly at the point where the topic is created. The topic can, of course, be changed at the cluster level.

However, in our setup, we derived certain streams to have a use-case based materialized-view and these materialized-view topics need to have unlimited retention time/bytes.

Describe the solution you'd like As a KSQLDB user, it would be nice to apply a configuration of a topic directly while creating CSAS or CTAS. For example, 

CREATE STREAM title_cased_songs WITH('log.retention.ms' = '-1') AS
    SELECT artist, UCASE(title) AS capitalized_song
    FROM rock_songs
    EMIT CHANGES;

Describe alternatives you've considered Indeed, configure the topic at the cluster-level is possible but leads to two open-ends. Moreover, integrating into CI/CD is even more complex.

MichaelDrogalis commented 4 years ago

@stefanfrehse Although it doesn't address all the aspects of your issue, you can set the retention time on a windowed materialized view using RETENTION N DAYS.

maksymilian-gasztych commented 3 years ago

@MichaelDrogalis Is there any estimate on when this feature could be released, if ever ? It would be really convenient to be able to set retention period related settings and cleanup.policy=compact,delete for tables.

MichaelDrogalis commented 3 years ago

Hey @maksymilian-gasztych. We're not working on this right now, but we'd take a patch for it if you have the bandwidth?