First let me say thanks for building a super useful platform big fan.
disclaimer - this may be a terrible idea, I am not across the full repository and its use cases.
Currently, incremental tables hold onto rows which are in the target, but are no longer in the source
This is because of the MERGE operation which is 'under the hood'
Although its not a common use case, it can happen a row is dropped from source data and we too want that row dropped in the target table
My proposed solution would be:
Allow for a config item called deleteNonMatched which is a boolean and only available for incremental tables
If this config item is set to true, then the compiler will add in the following line to the stored procedure
WHEN NOT MATCHED BY SOURCE THEN DELETE
Currently the only workaround solutions I have found it to use a post operation or abandon the Incremental table all together and use a custom operation instead.
Hey Dataform Team,
First let me say thanks for building a super useful platform big fan.
disclaimer - this may be a terrible idea, I am not across the full repository and its use cases.
My proposed solution would be:
Currently the only workaround solutions I have found it to use a post operation or abandon the Incremental table all together and use a custom operation instead.