customer-360 demo: error resulting in hidden system catalog [__databricks_internal]

[evogelpohl]

I attempted to install the c360 customer_360 demo on an Azure premium workspace w/ unity enabled. However, my Azure Quota for the clusters the demo wanted to create was insufficient to perform the demo install. It errored out w/ an azure-quota-error.

However, immediately after the error a system catalog appeared in Unity, [__databricks_internal], owned by system.

(correlated, perhaps not causal)

It contains a schema for a DLT project I created quite a long time ago & tables with the prefix [__dlt_materializationschema{guid}].

I can't re-hide this table. Thoughts?

[QuentinAmbard]

Hi, Non-metastore-admin users will not see it as long as permissions have not been explicitly granted. These catalogs/schemas are used to store internal tables for DLT, and will appear if DLT pipelines publishing to UC have been created in the workspace. if you drop the DLT pipeline I'd assume the schema should disappear by itself no?

[QuentinAmbard]

for your install you can run it with use_current_cluster = True, it'll prevent from creating a new cluster and avoid the quota issue

[evogelpohl]

right, but the catalog wasn't there prior to the installation of c360. Not sure why it appear other than that the c360 added DLT & exposed it. I had other DLT jobs running and it was hidden to me. I am a metastore owner. I'll drop the pipeline & see if it vanishes.

[QuentinAmbard]

I think it's just because it created a new DLT pipeline and the new one are using the internal catalog. If you create one your self it'll appear too.

[evogelpohl]

Thanks. Reminder on how those hidden tables tie to DLT.