search

databrickslabs / ucx

Automated migrations to Unity Catalog
Other
221 stars 78 forks source link

[FEATURE]: Support Microsoft Azure Government workspaces #2905

Open impactbee opened 1 day ago

impactbee commented 1 day ago

Is there an existing issue for this?

Problem statement

Workspaces in the Azure GovCloud regions do not have DBSQL, which translates to not having SQL Warehouses or Dashboards. UCX cannot be installed in these workspaces as a warehouse is required for installation and a dashboard is an artifact generated in the assessment flow.

Proposed Solution

A bared-down version of UCX where a SQL Warehouse is not needed and the workflow step creating a Dashboard is removed. The dashboards are replaced with notebooks containing the dashboard's queries and are run with an all-purpose cluster.

Additional Context

No response

nfx commented 15 hours ago

@impactbee theoretically this is possible:

  1. all SQL calls from CLI can be swapped via `CommandExecutionBackend' https://github.com/databrickslabs/lsql/blob/main/src/databricks/labs/lsql/backends.py#L227-L248
  2. dashboards can be executed via SQL notebooks
  3. there'll need to be additional catches of FeatureDisabled exceptions when we scan SQL warehouses

... but is this effort worth it? UC is not available in Azure Gov Cloud at the moment

impactbee commented 9 hours ago

@nfx I believe the effort is worth it for assessing the effort to migrate from non-UC workspaces to UC workspaces as well as migrating from Standard workspaces with no DBSQL option. These are workspaces that will not be upgraded in place.

For users that have to stay on GovCloud, AWS currently offers a better product experience so there are migrations from Azure to AWS and the UCX assessment allows them to plan their data migration better.

nfx commented 8 hours ago

@impactbee got it. so this feature won't be developed by the core UCX team (we don't have a govcloud test infra, hence we won't be able to support it), but i'm happy to guide the implementation team you staff to implement (and test) the necessary adjustments. please contact me on my Databricks email and cc the relevant account team and let's take it from there.