search

databrickslabs / ucx

Your best companion for upgrading to Unity Catalog. UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
Other
196 stars 70 forks source link

[BUG]: UCX failing with "Model serving is not enabled for your shard" #2082

Closed dgpoulet closed 6 days ago

dgpoulet commented 1 week ago

Is there an existing issue for this?

There's an existing issue: https://github.com/databrickslabs/ucx/issues/1228 which is marked as closed.

Current Behavior

Running UCX in Azure region UK South (which does not have Model Serving enabled) fails with the following error:

InternalError: Listing serving-endpoints failed: Model serving is not enabled for your shard. Please contact your organization admin or Databricks support.

Expected Behavior

Runs without error

Steps To Reproduce

No response

Cloud

Azure

Operating System

Windows

Version

latest via Databricks CLI

Relevant log output

InternalError: Listing serving-endpoints failed: Model serving is not enabled for your shard. Please contact your organization admin or Databricks support.

InternalError Traceback (most recent call last) File ~/.ipykernel/17807/command--1-3143761928:18 15 entry = [ep for ep in metadata.distribution("databricks_labs_ucx").entry_points if ep.name == "runtime"] 16 if entry: 17 # Load and execute the entrypoint, assumes no parameters ---> 18 entry[0].load()() 19 else: 20 import databricks_labs_ucx File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/databricks/labs/ucx/runtime.py:103, in main(*argv) 101 if len(argv) == 0: 102 argv = sys.argv --> 103 Workflows.all().trigger(*argv) File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/databricks/labs/ucx/runtime.py:80, in Workflows.trigger(self, *argv) 78 workflow = self._workflows[workflow_name] 79 if task_name == "parse_logs": ---> 80 return ctx.task_run_warning_recorder.snapshot() 81 # `{{parent_run_id}}` is the run of entire workflow, whereas `{{run_id}}` is the run of a task 82 workflow_run_id = named_parameters.get("parent_run_id", "unknown_run_id") File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/databricks/labs/ucx/installer/logs.py:203, in TaskRunWarningRecorder.snapshot(self) 201 error_messages.append(message) 202 if len(error_messages) > 0: --> 203 raise InternalError("\n".join(error_messages)) 204 return log_records InternalError: Listing serving-endpoints failed: Model serving is not enabled for your shard. Please contact your organization admin or Databricks support.
nfx commented 1 week ago

it looks like https://github.com/databrickslabs/ucx/pull/1618 didn't fully fix it