sodadata / soda-core

:zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io
https://go.soda.io/core-docs
Apache License 2.0
1.92k stars 209 forks source link

Contract support for staging envs #2164

Closed tombaeyens closed 1 month ago

tombaeyens commented 2 months ago

The reason that we should discuss this is that users ask for support of staging environments. We have to be able to explain to CEs how we are going to support it. Or at least have a prescriptive approach on how our features help with this use case. We may already have variables, but is that enough to support all the use cases we know around staging envs?

Assumption:

Alternative: Use the same file for both the production as well as the staging dataset.

As a producer, I want to run the use the same contract YAML file to verify data in a test environment as in the production environment

As a producer, I want to skip certain checks in test environment or in production environment

Idea: The contract could include test table values like sample data and a Soda tool could create the test table and insert the data.

tools-soda commented 2 months ago

CLOUD-8485

dirkgroenen commented 1 month ago

Moved https://sodadata.slite.com/api/s/RKhTABLUVmUy8Y/Representing-testing-staging-environments-in-Contracts