Some users have, on the order of, 20ish source tables that are defined in their source package. They have multiple copies of packages for kafka and local. That means they have about 50 .table.json configuration files. Most of the content in those files is identical: the flink connector configuration, the watermark, the metadata for kafka. Even the timestamp is mostly the same.
We can make those configuration files DRYer by having a root package-name.shared.json file where all the shared configuration is stored. On import, the table.json file gets merged with the .shared.json (with the former taking precedence) before reading it. That’s exactly the same thing we do with package.json files and the infrastructure exists already.
Some users have, on the order of, 20ish source tables that are defined in their source package. They have multiple copies of packages for kafka and local. That means they have about 50
.table.json
configuration files. Most of the content in those files is identical: the flink connector configuration, the watermark, the metadata for kafka. Even the timestamp is mostly the same.We can make those configuration files DRYer by having a root
package-name.shared.json
file where all the shared configuration is stored. On import, the table.json file gets merged with the .shared.json (with the former taking precedence) before reading it. That’s exactly the same thing we do with package.json files and the infrastructure exists already.