databricks-demos / dbdemos

Demos to implement your Databricks Lakehouse
Other
255 stars 80 forks source link

CLUSTER_CREATION_ERROR when installing demos on Workspaces with the Compliance Security Profile enabled #17

Closed rodrigoimprota-db closed 1 year ago

rodrigoimprota-db commented 1 year ago

When installing a dbdemo in a HIPAA-compliant environment, the following error is thrown, and the cluster is not created.

WARN: couldn't create the cluster for the demo: {'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'Workspace restricted to instance types that encrypt in transit. Please specify one such driver node type'}

It happened when I tried to install the delta-lake demo on a HIPAA-compliant workspace on AWS that enforces Nitro instance types.

Code:

%pip install dbdemos

import dbdemos
dbdemos.list_demos()

dbdemos.install('delta-lake')

The notebooks are created without problems, and the demo runs smoothly in a user-created cluster.

QuentinAmbard commented 1 year ago

Thanks Rordrigo for pointing that, we'll try to improve that and fallback using the current cluster when I can.

QuentinAmbard commented 1 year ago

Hi @rodrigoimprota-db , we added the option to run the demo with your current cluster. We now fail with a better error, suggesting you add the use_current_cluster = True option. It'll use your current cluster instead of trying to create a new one. Longer term we'll likely check if your current cluster respect the demo requirement (mostly DBR runtime + UC) and use it by default instead of creating a new one

image

I'm closing this issue, please re-open if this doesn't work for you.