search

databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
255 stars 80 forks source link

DLT - Retail Data Quality Stats throws errors #114

Open bicaluv opened 2 months ago

bicaluv commented 2 months ago

I installed DLT Loans demo using dbdemos.install('dlt-loans', use_current_cluster = True).
Navigated to the DLT - Retail Data Quality Stats dashboard but only get the following errors on the different panes.
BTW: we do not use Hive metastore in our workspace. We use UC only.

[TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dbdemos`.`dlt_expectations` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; line 2, pos 123
QuentinAmbard commented 2 months ago

We have a script to create this table, in the notebook. You can check the companion notebook it'll write the content to the table.

bicaluv commented 2 months ago

would you mind directing me to the right notebook?
And will I have to make additional changes when the pipeline storage options ares changed to use UC?

In the DLT - Retail Data Quality Stats notebook, there are other notebooks mentioned, like 02.1-Delta-Live-Table-Ingestion and 02.3-DLT-expectation-dashboard-data-prep which I'm note able to find.

bicaluv commented 2 months ago

Is it that simple to only change the dashboards queries to use table playground.dbdemos.dlt_expectations ? (playground is the catalog where I configured the DLT pipeline)
Nope, it's not. The table is generated with fake data using notebook 01-load-data-quality-dashboard. But how to I get the real data coming from expectation events when using UC?

bicaluv commented 2 months ago

Let me answer my question: use SELECT * FROM event_log(TABLE(playground.xxxx.cleaned_new_txs));
see https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/observability