Can't dbx execute on Shared compute clusters

[danricbede]

Expected Behavior

Able to run dbx execute on a shared compute cluster.

Current Behavior

Dbx throws error when installing packages on cluster: CalledProcessError: Command 'pip --disable-pip-version-check install --force-reinstall "/dbfs/dbx/mywheel.whl[test]"' returned non-zero exit status 1.

Steps to Reproduce (for bugs)

1. Set up a cluster using Shared compute
2. Attempt to run dbx execute <workflow-name> --task <task-name>

Context

Additional output:

[dbx][2023-08-07 15:57:23.792] ✅ Uploading package - done
send: b'POST /api/1.2/commands/execute HTTP/1.1\r\nHost: <removed>.azuredatabricks.net\r\nuser-agent: databricks-cli-0.17.7-cicdtemplates-\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nAuthorization: Bearer <removed>-2\r\nContent-Type: text/json\r\nContent-Length: 271\r\n\r\n'
send: b'{"language": "python", "clusterId": "<removed>", "contextId": "<removed>", "command": "%pip install --force-reinstall \\"/dbfs/dbx/<removed>-0.1.0-py3-none-any.whl[test]\\""}'

...

⠙ Installing package on the cluster 📦[dbx][2023-08-07 15:57:40.732] Execution failed, please follow the given error
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
File ~/.ipykernel/1198/command--1-2089833704:1
----> 1 get_ipython().run_line_magic('pip', 'install --force-reinstall "/dbfs/dbx/<removed>-0.1.0-py3-none-any.whl[test]"')

File /databricks/python/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2369, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2367     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2368 with self.builtin_trap:
-> 2369     result = fn(*args, **kwargs)
   2371 # The code below prevents the output from being displayed
   2372 # when using magics with decodator @output_can_be_silenced
   2373 # when the last Python token in the expression is a ';'.
   2374 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File /databricks/python_shell/dbruntime/PipMagicOverrides.py:35, in PipMagicOverrides.pip(self, line)
     33 @line_magic
     34 def pip(self, line):
---> 35     self.pipMagicHandler.runCmd("pip", line)

File /databricks/python_shell/dbruntime/PipMagicOverrides.py:61, in PipMagicHandler.runCmd(self, magicCmd, line)
     59     print(PYTHON_RESTART_WARNING)
     60 if parsedResult.rewrittenCommand():
---> 61     self.executePipCommand(parsedResult)
     62 envManager.postExecute(parsedResult)
     63 if parsedResult.isMutation():
     64     # double print this output is at the end so it is more
     65     # likely to be seen

File /databricks/python_shell/dbruntime/PipMagicOverrides.py:122, in PipMagicHandler.executePipCommand(self, result)
    120     sys.stdout.flush()
    121     if returncode != 0:
--> 122         raise subprocess.CalledProcessError(returncode, origCmd)
    123 finally:
    124     end = time.time()

CalledProcessError: Command 'pip --disable-pip-version-check install --force-reinstall "/dbfs/dbx/<removed>-0.1.0-py3-none-any.whl[test]"' returned non-zero exit status 1.

When I run directly in a notebook, the error is expanded upon:

Note: you may need to restart the kernel using dbutils.library.restartPython() to use updated packages.
WARNING: Requirement '/dbfs/dbx/<removed>-0.1.0-py3-none-any.whl' looks like a filename, but the file does not exist
Processing /dbfs/dbx/<removed>-0.1.0-py3-none-any.whl
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/dbfs/dbx/<removed>-0.1.0-py3-none-any.whl'

CalledProcessError: Command 'pip --disable-pip-version-check install --force-reinstall "/dbfs/dbx/<removed>-0.1.0-py3-none-any.whl"' returned non-zero exit status 1.

The same command works when the access mode is Single User or Unrestricted.

Your Environment

• dbx version used: 0.8.18
• Databricks Runtime version: 13.2, 13.3

[danricbede]

After some more digging I came across this page.

On Databricks Runtime 13.1 and above, libraries that are referenced using DBFS filepaths are not supported on clusters that use shared access mode in a Unity Catalog-enabled workspace, whether in the DBFS root or an external location mounted to DBFS.

This also occurs when doing a dbx launch, not just execute.

I will need to figure something out - UC complicates things somewhat.