Is your feature request related to a problem? Please describe.
Dictionaries that are backed by large clickhouse tables/queries typically are reloaded from the entire dataset. This may be inefficient if only a few rows have changed. There is an option to specify a field that enables new data to be identified, thereby minimizing the updated data.
Describe the solution you'd like
Support the update_field syntax in the dictionary materialization
SOURCE(CLICKHOUSE(...
update_field 'added_time'
QUERY '
SELECT my_arr.1 AS x, my_arr.2 AS y, creation_time
FROM (
SELECT arrayZip(x_arr, y_arr) AS my_arr, creation_time
FROM dictionary_source
WHERE {condition}
)'
))
Is your feature request related to a problem? Please describe. Dictionaries that are backed by large clickhouse tables/queries typically are reloaded from the entire dataset. This may be inefficient if only a few rows have changed. There is an option to specify a field that enables new data to be identified, thereby minimizing the updated data.
Describe the solution you'd like Support the
update_fieldsyntax in the dictionary materializationAdditional context Add any other context or screenshots about the feature request here.