search

databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
255 stars 80 forks source link

Why do I need 16 cores or more for a DEMO? #122

Open jhrobinson opened 1 month ago

jhrobinson commented 1 month ago

I have received the following error message more than once. Even when I use the trial version, I get the same error message. Why do I need so many cores for a demo?

Summary 2-10 Workers 64-320 GB Memory 16-80 Cores 1 Driver 32 GB Memory, 8 Cores Runtime 13.3.x-cpu-ml-scala2.12 Unity Catalog Standard_D8ds_v4 4–17 DBU/h Azure Quota Exceeded Exception: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 11, Current Usage: 4, Additional Required: 16, (Minimum) New Limit Required: 20. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%228b64f0d6-e1f2-4deb-9b69-cb68ec623778%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22cores%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:20,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22cores%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed.

QuentinAmbard commented 1 month ago

thanks for flagging this, which demo is doing that? You can run it with use_current_cluster = True to avoid the issue while we check the demo setup.

jhrobinson commented 1 month ago

Two of them: dbdemos-lakehouse-fsi-credit dbdemos-lakehouse-retail-c360.

How do I set use_current_cluster = true?

QuentinAmbard commented 1 month ago

just in the install function, you can run dbdemos.help() It'll prevent from starting a new cluster

jhrobinson commented 1 month ago

So like this?

dbdemos.help(use_current_cluster = true)

QuentinAmbard commented 1 month ago

dbdemos.install("lakehouse-retail-c360", use_current_cluster = true)

jhrobinson commented 1 month ago

I added the code you wrote above. No errors, but shouldn't there be a new schema added to UC?

jhrobinson commented 1 month ago

Is this demo created in such a way that you can't deviate at all? It seems that there are all kinds of issues if you can't use a certain workspace or cluster.

QuentinAmbard commented 1 month ago

@jhrobinson the install will create a job and use your current cluster to run the job, the job will create a schema/catalog and save the data here. You can check the job for more details you have the link in the install report. Make sure your cluster is ML runtime + UC enable (single user is probably easier)