Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
Is your feature request related to a problem? Please describe.
Currently the SynchronizeDeltaToSnowflakeTask implements merge logic for Snowflake target. Because the merge configuration is injected inside the task itself it becomes difficult to customize it. Also the task itself is overloaded with logic and function.
Describe the solution you'd like
I would like to decouple merge logic from Task itself and have a separated SnowflakeMergeWriter which can be passed to the task if needed.
Describe alternatives you've considered
Keep as it is and make users to override all custom logic.
Is your feature request related to a problem? Please describe.
Currently the SynchronizeDeltaToSnowflakeTask implements merge logic for Snowflake target. Because the merge configuration is injected inside the task itself it becomes difficult to customize it. Also the task itself is overloaded with logic and function.
Describe the solution you'd like
I would like to decouple merge logic from Task itself and have a separated
SnowflakeMergeWriterwhich can be passed to the task if needed.Describe alternatives you've considered
Keep as it is and make users to override all custom logic.