sodadata / soda-core

:zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io
https://go.soda.io/core-docs
Apache License 2.0
1.91k stars 208 forks source link

Contract support for staging envs #2164

Closed tombaeyens closed 1 month ago

tombaeyens commented 1 month ago

The reason that we should discuss this is that users ask for support of staging environments. We have to be able to explain to CEs how we are going to support it. Or at least have a prescriptive approach on how our features help with this use case. We may already have variables, but is that enough to support all the use cases we know around staging envs?

Assumption:

Alternative: Use the same file for both the production as well as the staging dataset.

As a producer, I want to run the use the same contract YAML file to verify data in a test environment as in the production environment

As a producer, I want to skip certain checks in test environment or in production environment

Idea: The contract could include test table values like sample data and a Soda tool could create the test table and insert the data.

tools-soda commented 1 month ago

CLOUD-8485

dirkgroenen commented 1 month ago

Moved https://sodadata.slite.com/api/s/RKhTABLUVmUy8Y/Representing-testing-staging-environments-in-Contracts