Failed to import Personal Compute Cluster

[danny-db]

I'm trying to run an import pipeline and there is an error regarding Personal Compute Cluster creation.

Creating cluster: Danny Wong's Personal Compute Cluster post: https://adb-6547519379824786.6.azuredatabricks.net/api/2.0/clusters/create 2022-11-20,17:35:54;WARNING;{"error_code": "INVALID_PARAMETER_VALUE", "message": "Validation failed for data_security_mode, the value must be present"}

2022-11-20,17:35:54;ERROR;{"error_code": "INVALID_PARAMETER_VALUE", "message": "Validation failed for data_security_mode, the value must be present", "http_status_code": 400} {'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'Validation failed for data_security_mode, the value must be present', 'http_status_code': 400} 2022-11-20,17:35:54;INFO;1120-061854-8iikno1m found in checkpoint Get: https://adb-6547519379824786.6.azuredatabricks.net/api/2.0/clusters/list The old cluster id 1120-060216-vcqgev47 with the original_creator of danny.wong@databricks.com does not get logged for EditClusterOwner due to some problems. Applying acl for Danny Wong's Personal Compute Cluster 2022-11-20,17:35:55;INFO;Skipping inherited permissions.. Get: https://adb-6547519379824786.6.azuredatabricks.net/api/2.0/clusters/list Traceback (most recent call last): File "/Users/danny.wong/Dev/migrate/migration_pipeline.py", line 328, in main() File "/Users/danny.wong/Dev/migrate/migration_pipeline.py", line 324, in main pipeline.run() File "/Users/danny.wong/Dev/migrate/pipeline/pipeline.py", line 64, in run future.result() File "/opt/homebrew/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 446, in result return self.get_result() File "/opt/homebrew/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 391, in get_result raise self._exception File "/opt/homebrew/anaconda3/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/Users/danny.wong/Dev/migrate/pipeline/pipeline.py", line 73, in _run_task task.run() File "/Users/danny.wong/Dev/migrate/tasks/tasks.py", line 227, in run cl_c.import_cluster_configs() File "/Users/danny.wong/Dev/migrate/dbclient/ClustersClient.py", line 297, in import_cluster_configs raise ValueError(error_message)

[gregwood-db]

@danny-db would you mind providing the full error logs, as well as the command you used to run this? Is this a UC-enabled WS?

[danny-db]

Attached please find the full logs.

Command used for export: python3 migration_pipeline.py --profile sourceWS --export-pipeline --use-checkpoint --azure

Command used for import: python3 migration_pipeline.py --profile destWS --import-pipeline --use-checkpoint --azure --session M20221120174324

These are not UC-enabled workspaces. These are 2 brand new workspaces created for this testing purpose, and on the source workspace I've a personal compute cluster created. It failed to import after the export success with warnings. These 2 workspaces are on Azure.

On Tue, Nov 29, 2022 at 2:08 AM gregwood-db @.***> wrote:

@danny-db https://github.com/danny-db would you mind providing the full error logs, as well as the command you used to run this? Is this a UC-enabled WS?

— Reply to this email directly, view it on GitHub https://github.com/databrickslabs/migrate/issues/215#issuecomment-1329265413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY5R5YPRG5A3BJQNWPU4RPDWKTDGJANCNFSM6AAAAAASFWD4RU . You are receiving this because you were mentioned.Message ID: @.***>

--

Danny Wong

Senior Solutions Architect

+61 401 838 643

@.***

AU Public Sector Meetup Group https://www.meetup.com/databricks-public-sector-user-group/

[gregwood-db]

@danny-db this should be corrected in https://github.com/databrickslabs/migrate/commit/6f488bdec6a7ce7a10c9418b6ec6382e6de926cc (we were ignoring the data_security_mode attribute during cluster export)