databrickslabs / migrate

Old scripts for one-off ST-to-E2 migrations. Use "terraform exporter" linked in the readme.
Other
186 stars 128 forks source link

Default cluster configurations don't support HIPAA customers on AWS #270

Closed jeff-shuttNH closed 1 year ago

jeff-shuttNH commented 1 year ago

While importing into an AWS workspace that has the HIPAA module enabled, the default cluster configurations used during import do not support EC2 instances that have Nitro support. This results in getting an error like the following:

2023-06-08,22:12:02;INFO;Starting cluster with name: Workspace_Migration_Work_Leave_Me_Alone
2023-06-08,22:12:02;DEBUG;https://dbc-123-abc.cloud.databricks.com:443 "POST /api/2.0/clusters/create HTTP/1.1" 400 None
2023-06-08,22:12:02;WARNING;{"error_code": "INVALID_PARAMETER_VALUE", "message": "Workspace restricted to instance types that encrypt in transit. Please specify one such driver node type", "details": [{"@type": "type.googleapis.com/google.rpc.ErrorInfo", "reason": "CM_API_ERROR_SOURCE_CALLER_ERROR", "domain": ""}]}

The following table breaks down some recommended changes that should work based on which instance types support Nitro.

File Current Instance Type Proposed Instance Type
aws_cluster_table_acls.json m4.xlarge m5.xlarge
aws_cluster.json i3.xlarge i4i.xlarge
default_jobs_cluster_aws.json i3.xlarge i4i.xlarge
gregwood-db commented 1 year ago

Thanks @jeff-shuttNH - I'll add a -hipaa flag to the import options to allow the alternate instances.

gregwood-db commented 1 year ago

Flag added via https://github.com/databrickslabs/migrate/commit/af01d06efda3f5146138b6fdc6de80ed05fb219e