databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
255 stars 80 forks source link

error while running dbdemos.install #117

Closed srbbhat closed 2 months ago

srbbhat commented 2 months ago

I am seeing an error while running dbdemos.install

'''PERMISSION_DENIED: Access denied. Cause: AccessDeniedException error from cloud storage provider. exceptionTraceId=28e27f64-d714-4beb-b691-b39cfe310267'''

dbdemos.install('uc-04-system-tables', use_current_cluster=True) or dbdemos.install('uc-04-system-tables') or also tried to run with custom catalog as well

dbdemos.install('uc-04-system-tables', catalog='dev_bronze', schema='billing', path='/Workspace/Users/sbhat@breadfinance.com/', overwrite=True, use_current_cluster=True )

here is the status of system tables: {"schemas":[{"schema":"storage","state":"ENABLE_COMPLETED"},{"schema":"access","state":"ENABLE_COMPLETED"},{"schema":"billing","state":"ENABLE_COMPLETED"},{"schema":"compute","state":"ENABLE_COMPLETED"},{"schema":"marketplace","state":"ENABLE_COMPLETED"},{"schema":"operational_data","state":"UNAVAILABLE"},{"schema":"lineage","state":"UNAVAILABLE"},{"schema":"information_schema","state":"ENABLE_COMPLETED"}]}%

image

https://bd-bd-ue2-npp-databricks.cloud.databricks.com/jobs/767426908881482/runs/86449372514313?o=2944657783116080

image

QuentinAmbard commented 2 months ago

thanks I'll have a look

srbbhat commented 2 months ago

@QuentinAmbard do you have an update?

QuentinAmbard commented 2 months ago

did you check the actual SQL error following the link you have in the message? I'd say that your workspace isn't properly setup with UC (permission issue with the cloud storage?)

srbbhat commented 2 months ago

I don't see any permission issue, create schema was successful, CREATE TABLE has failed. I do have the right permissions. this catalog is UC enabled. do you think this is related to the way Managed Storage is setup? We use External Storage for other UC tabled tables

--param query - the import needs the query to be run first even empty. CREATE SCHEMA IF NOT EXISTS dev_bronze.billing_forecast; CREATE TABLE IF NOT EXISTS dev_bronze.billing_forecast.billing_forecast (sku STRING, workspace_id STRING);

PERMISSION_DENIED: Access denied. Cause: AccessDeniedException error from cloud storage provider. exceptionTraceId=b28e2f64-b232-49ef-9908-f92e277e1b39

QuentinAmbard commented 2 months ago

you have a permission issue for the dev_bronze catalog from what you shared: PERMISSION_DENIED: Access denied. Cause: AccessDeniedException error from cloud storage provider. exceptionTraceId=b28e2f64-b232-49ef-9908-f92e277e1b39

It's usually the managed identity not properly set to the storage, or something similar

srbbhat commented 2 months ago

the issue has been resolved! I am able figure it out ! Addition of LOCATION to the create table seems to have worked!

srbbhat commented 2 months ago

closing