Clarify batch testing strategies

[tombaeyens]

The reason I think we should discuss is that there is a problem in the current approach. I think in our current approach we don't provide the necessary transparency to users and in pipeline tests may leave records that are not tested. We also have to make sure that the unified concept includes all the necessary scheduleing and triggering that we need so that our CEs don't have to start building around it.

If we want a unified platform and a prescriptive approach, then we need to ensure that our vision includes each of the strategies our customers need.

This is related to the scheduleing discussion.

Background:

• https://discourse.getdbt.com/t/on-the-limits-of-incrementality/303
• https://medium.com/geekculture/idempotent-data-pipeline-with-dbt-and-airflow-597d6f65a811
• https://techblog.finatext.com/dbt-incremental-strategy-and-idempotency-877993f48448

Also to be considered as part of this is backfilling. We know our CEs have to answer questions on this topic. How will our unified solution in the end look like when customers ask for backfilling?

[tools-soda]

CLOUD-8490

[dirkgroenen]

Moved https://sodadata.slite.com/api/s/JSEdSB5RuxcRwR/Batch-testing-strategy