search

elementary-data / elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
https://www.elementary-data.com/
Apache License 2.0
1.88k stars 159 forks source link

Allow anomalies test on nested fields #903

Open JavierLopezT opened 1 year ago

JavierLopezT commented 1 year ago

Right now, anomalies tests on nested fields (for instance Struct fields on BigQuery) are not supported, but it's something I definitely need, since a lot of important metrics are stored in struct fields.

Thanks to @Maayan-s I now know that that is because elementary has a flow in the test that checks that the column actually exists and what is it's data type to avoid errors. One possible solution, as suggested by her, would be to create a new nested_field_anomalies test that would just avoid the steps of checking if the column exists and what's it's data type.

I am open to work on this providing that the change is not very time consuming

k-lawecki commented 9 months ago

We would love this too!

geo909 commented 2 months ago

It would be great if this was implemented; we use structs heavily and I cannot use elementary in our models without resorting to workarounds.

Edit: I have no idea what's going on, but my comment triggered some github action, which failed: https://github.com/elementary-data/elementary/actions/runs/9892357654