[FEATURE]: DBR upgrade mechanism

[chase-edwards-db]

There is no systematic way to upgrade DBR. Often, the first step large customers need to take is a DBR upgrade to move to a UC-supported runtime. Note that this feature could (and should) be built for the generic case, independent of UC migrations as historically it has always been a field need.

Related issues:

• [ ] https://github.com/databrickslabs/ucx/issues/1124
• [ ] #1122

Proposed Solution

The feature should:

• Evaluate and display unsupported DBR impact preventing UC migration
• Migrate clusters and associated policies to new target runtime
• Validate successful migration and associated workloads.

This can/should be rolled into #1122, where validation should be done via:

• Successful change of targeted cluster configuration.
• Successful execution of targeted workflows.

and should surface the following information:

• Errors upon failure of DBR upgrade
• Migration rollup success metrics (% clusters migrated successfully, % migrated jobs executed successfully, % migrated workflows executed successfully)
• Tables with item-level success metrics e.g. for clusters (cluster_id | migrated_flag | error_message) and for workflows (job_id, execution_success, error_message)

Additional Context

For one enterprise customer, it took nearly 12 months to move from 7.3LTS to 9.1LTS.

[nfx]

@chase-edwards-db how can validation be automated? Please describe

[chase-edwards-db]

Updated, let me know if more info needed @nfx

[pohlposition]

This seems like a HUGE feature request. Like it could be it's own Labs project. I'm not suggesting it's not important, just a large amount of work; and somewhat independent to the goals of UCX.

[chase-edwards-db]

@pohlposition completely agree. I was hoping to use this as a catalyst, but I suppose as Serverless adoption/usage accelerates there may not be a need in a few years. Maybe a "lighter" version to support UC migrations, specifically?

[chase-edwards-db]

For ref, we do have this (per @nfx's ask on upgrade documentation): https://dbrmg.databricks.com/?cloud_provider=aws So, it would be more about applying translation of these, or UC-specific changes (not clearly documented from what I can see).