I was interested in making a custom materialization that would be similar to incremental by time range and by partition.
Everything seems to be there to do it, except a few properties to enable the chunking.
Would it be possible to add these properties to the custom kind?
batch_size: t.Optional[SQLGlotPositiveInt] = None
batch_concurrency: t.Optional[SQLGlotPositiveInt] = None
lookback: t.Optional[SQLGlotPositiveInt] = None
If the properties were defined to pull the values out of materializationproperties would it work simply without breaking other custom materializations that don't set them?
izeigerman
commented
5 months ago
I was interested in making a custom materialization that would be similar to incremental by time range and by partition. Everything seems to be there to do it, except a few properties to enable the chunking.
Would it be possible to add these properties to the custom kind? batch_size: t.Optional[SQLGlotPositiveInt] = None batch_concurrency: t.Optional[SQLGlotPositiveInt] = None lookback: t.Optional[SQLGlotPositiveInt] = None
https://github.com/TobikoData/sqlmesh/blob/034476e7f64d261860fd630c3ac56d8a9c9f3e3a/sqlmesh/core/model/kind.py#L766
If the properties were defined to pull the values out of materializationproperties would it work simply without breaking other custom materializations that don't set them?