search

DataSQRL / sqrl

Flexible development framework for building streaming data applications in SQL with Kafka, Flink, Postgres, GraphQL, and more.
https://www.datasqrl.com/
96 stars 13 forks source link

Support a root configuration file #924

Open henneberger opened 1 week ago

henneberger commented 1 week ago

Some users have, on the order of, 20ish source tables that are defined in their source package. They have multiple copies of packages for kafka and local. That means they have about 50 .table.json configuration files. Most of the content in those files is identical: the flink connector configuration, the watermark, the metadata for kafka. Even the timestamp is mostly the same.

We can make those configuration files DRYer by having a root package-name.shared.json file where all the shared configuration is stored. On import, the table.json file gets merged with the .shared.json (with the former taking precedence) before reading it. That’s exactly the same thing we do with package.json files and the infrastructure exists already.

henneberger commented 1 week ago

Can we punt on this in favor of supporting flink create table statements?