starlake-ai / starlake

Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
http://starlake.ai/
Apache License 2.0
44 stars 20 forks source link

[FEATURE] - Add CLI assertion for tables #967

Open tiboun opened 1 month ago

tiboun commented 1 month ago

Today, assertions are made within the framework of the load. This link prevents assertions from being used in a scope other than that of the loaded table.

In fact, tables from the same domain can be loaded into N tables in parallel. To ensure the integrity of these tables, it is necessary to be executed after all loads.

The creation of a command, similar to the load command, to verify assertions would enable this kind of verification, for example in the case of an exploded Excel file.