[BUG]: TypeError: cannot pickle '_thread.lock' object

CedricEdouardKassi commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

Hello, I'm facing with this issue when trying to create databricks account group from local Databricks group with the following command : databricks labs ucx create-account-groups --workspace-ids --debug -p

I got this error:

TypeError: cannot pickle '_thread.lock' object

For information, I'm megastore admin and Databricks account admin.

Expected Behavior

No response

Steps To Reproduce

databricks labs ucx create-account-groups --workspace-ids --debug -p

Cloud

Azure

Operating System

macOS

Version

latest via Databricks CLI

Relevant log output

ERROR [d.l.ucx.create-account-groups] Failed to call create-account-groups: Traceback (most recent call last):
  File "/Users/cedric/.databricks/labs/ucx/state/venv/lib/python3.11/site-packages/databricks/labs/blueprint/cli.py", line 113, in _route
    cmd.fn(**kwargs)
  File "/Users/cedric/.databricks/labs/ucx/lib/src/databricks/labs/ucx/cli.py", line 122, in create_account_groups
    ctx.account_workspaces.create_account_level_groups(prompts, ctx.workspace_ids)
  File "/Users/cedric/.databricks/labs/ucx/lib/src/databricks/labs/ucx/account/workspaces.py", line 82, in create_account_level_groups
    all_valid_workspace_groups = self._get_valid_workspaces_groups(prompts, workspace_ids)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cedric/.databricks/labs/ucx/lib/src/databricks/labs/ucx/account/workspaces.py", line 190, in _get_valid_workspaces_groups
    self._load_workspace_groups(prompts, workspace, all_workspaces_groups)
  File "/Users/cedric/.databricks/labs/ucx/lib/src/databricks/labs/ucx/account/workspaces.py", line 195, in _load_workspace_groups
    client = self.client_for(workspace)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cedric/.databricks/labs/ucx/lib/src/databricks/labs/ucx/account/workspaces.py", line 30, in client_for
    return self._ac.get_workspace_client(workspace)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cedric/.databricks/labs/ucx/state/venv/lib/python3.11/site-packages/databricks/sdk/__init__.py", line 929, in get_workspace_client
    config = self._config.deep_copy()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cedric/.databricks/labs/ucx/state/venv/lib/python3.11/site-packages/databricks/sdk/config.py", line 526, in deep_copy
    return copy.deepcopy(self)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 238, in _deepcopy_method
    return type(x)(x.__func__, deepcopy(x.__self__, memo))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.8/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle '_thread.lock' object

nfx commented 1 month ago

looks like an SDK problem

arqgroup-charunathan commented 1 month ago

Hi @nfx / @CedricEdouardKassi , I'm also facing the same issue while executing the same ucx command. Please let me know if you have fixed this issue.

nikhilmish-db commented 1 month ago

We are also facing this issue, and CX is waiting for a resolution. 16:28:19 ERROR [src/databricks/labs/ucx.create-account-groups] TypeError: cannot pickle '_thread.lock' object

cekowalski72 commented 1 month ago

One of my customers is also facing this issue when running "databricks labs ucx sync-workspace-info --profile accounts"

nfx commented 1 month ago

depends on a bug in SDK

qziyuan commented 1 month ago

So any code that calls AccountClient.get_workspace_client with SDK v0.29.0 will have this error. With that, following commands in UCX v0.28.0 and above will suffer from this issue:

create-account-groups command
sync-workspace-info command
join-collection command
ensure-assessment-run command
report-account-compatibilitycommand
assign-metastore command, if default_catalog parameter is set.
UCX installation will fail if has UCX_FORCE_INSTALL=account

Here is a workaround:

The idea is installing a separate older version UCX bins (v0.23.0) in the local laptop. We only need the UCX bins and older version SDK which does not have the above bug, so no need to finish the workspace installation part (deploy workspace config, inventory database, workflows etc.)

UCX and all it's dependencies, including the SDK, will be installed in $HOME/.databricks/labs/ucx/. So we need to set new $HOME temporarily, like export HOME=/Users/ziyuan.qin/ucx_v23. So the older UCX installed later will be separated from the default existing current version UCX.
Install UCX v0.23.0: databricks labs install ucx@v0.23.0 -p <profile>. Answer no when asked "[ADVANCED] UCX is already installed on this workspace. Do you want to create a new installation?", because just need older UCX and SDK bin to be installed in the laptop, and we don't need following installation and deployment in workspace.
Now you can retry those UCX commands failed with _thread.lock error.
By setting $HOME env, you can control which version of UCX bins to be used.
With old version UCX, you need to databricks auth login to all workspaces that your command will be executed against, just databricks auth login to account console won't be enough. For example:
- sync-workspace-info store all workspace id and name under the account, so you need to databricks auth login to all the workspaces.
- For create-account-groups command, you need to databricks auth login to all the workspaces listed in --workspace-ids.

qziyuan commented 1 month ago

ES-1203418 is filed to fix the SDK bug.

tanyapohn commented 1 month ago

So any code that calls AccountClient.get_workspace_client with SDK v0.29.0 will have this error. With that, following commands in UCX v0.28.0 and above will suffer from this issue:

create-account-groups command

sync-workspace-info command

join-collection command

ensure-assessment-run command

report-account-compatibilitycommand

assign-metastore command, if default_catalog parameter is set.

UCX installation will fail if has UCX_FORCE_INSTALL=account

Here is a workaround:

The idea is installing a separate older version UCX bins (v0.23.0) in the local laptop. We only need the UCX bins and older version SDK which does not have the above bug, so no need to finish the workspace installation part (deploy workspace config, inventory database, workflows etc.)

UCX and all it's dependencies, including the SDK, will be installed in $HOME/.databricks/labs/ucx/. So we need to set new $HOME temporarily, like export HOME=/Users/ziyuan.qin/ucx_v23. So the older UCX installed later will be separated from the default existing current version UCX.

Install UCX v0.23.0: databricks labs install ucx@v0.23.0 -p <profile>. Answer no when asked "[ADVANCED] UCX is already installed on this workspace. Do you want to create a new installation?", because just need older UCX and SDK bin to be installed in the laptop, and we don't need following installation and deployment in workspace.

Now you can retry those UCX commands failed with _thread.lock error.

By setting $HOME env, you can control which version of UCX bins to be used.

With old version UCX, you need to databricks auth login to all workspaces that your command will be executed against, just databricks auth login to account console won't be enough. For example:

sync-workspace-info store all workspace id and name under the account, so you need to databricks auth login to all the workspaces.

For create-account-groups command, you need to databricks auth login to all the workspaces listed in --workspace-ids.

I'd followed step no. 1 and 2. After I ran databricks labs install ucx@v0.23.0 -p <profile>, it doesn't prompt the question but raises the error said:

Traceback (most recent call last):
  File "/Users/tanyapohn.p/ucx_v23/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 49, in <module>
    from databricks.labs.ucx.contexts.account_cli import AccountContext
  File "/Users/tanyapohn.p/ucx_v23/.databricks/labs/ucx/lib/src/databricks/labs/ucx/contexts/account_cli.py", line 6, in <module>
    from databricks.labs.ucx.account.aggregate import AccountAggregate
  File "/Users/tanyapohn.p/ucx_v23/.databricks/labs/ucx/lib/src/databricks/labs/ucx/account/aggregate.py", line 14, in <module>
    from databricks.labs.ucx.contexts.workspace_cli import WorkspaceContext
  File "/Users/tanyapohn.p/ucx_v23/.databricks/labs/ucx/lib/src/databricks/labs/ucx/contexts/workspace_cli.py", line 16, in <module>
    from databricks.labs.ucx.contexts.application import CliContext
  File "/Users/tanyapohn.p/ucx_v23/.databricks/labs/ucx/lib/src/databricks/labs/ucx/contexts/application.py", line 37, in <module>
    from databricks.labs.ucx.source_code.jobs import WorkflowLinter
  File "/Users/tanyapohn.p/ucx_v23/.databricks/labs/ucx/lib/src/databricks/labs/ucx/source_code/jobs.py", line 14, in <module>
    from databricks.labs.ucx.mixins.wspath import WorkspacePath
  File "/Users/tanyapohn.p/ucx_v23/.databricks/labs/ucx/lib/src/databricks/labs/ucx/mixins/wspath.py", line 9, in <module>
    from pathlib import Path, _PosixFlavour  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: cannot import name '_PosixFlavour' from 'pathlib' (/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/pathlib.py)
Error: installer: exit status 1

I am not quite sure what I missed

tanyapohn commented 1 month ago

For anyone who faces the same issue as I do when trying to work around, I figured it out the error above produces when installing v0.23.0. This can be fixed by installing v0.23.1 instead since _PosixFlavour had been resolved in v0.23.1 releases

ouranos commented 1 month ago

I'm having the same error on ucx@0.32.0 when running the sync-workspace-info command:

ERROR [d.l.ucx.sync-workspace-info] TypeError: cannot pickle '_thread.lock' object

Any update on this?

qziyuan commented 1 month ago

The upstream bug fix is being tracked in this ES ticket https://databricks.atlassian.net/browse/ES-1203418

@ouranos meanwhile you can try the workaround above to run sync-workspace-info, or instead use manual-workspace-info

ouranos commented 1 month ago

The upstream bug fix is being tracked in this ES ticket databricks.atlassian.net/browse/ES-1203418

This seems to be private so I cannot track the progress.

@ouranos meanwhile you can try the workaround above to run sync-workspace-info, or instead use manual-workspace-info

I've used manual-workspace-info for now.

sagpra commented 3 weeks ago

No proper ucx build is available till now. For every version there is some issues. Others are also getting same issues?

qziyuan commented 3 weeks ago

To whom do not have access to https://databricks.atlassian.net/browse/ES-1203418, here is the update: A PR is created to fix the cannot pickle '_thread.lock'. However it will cause a regression that each workspace your command needs to access has to be authenticated in your machine first. The original goal is you authenticate to account once, and then be able to connect to all the workspace you have access to. We still need a better fix. Stay tuned.

qziyuan commented 2 weeks ago

No proper ucx build is available till now. For every version there is some issues. Others are also getting same issues?

@sagpra My customer tried today and the workaround is still working for them. I'm not sure what is the issue in your case. If you are blocked by this, one possible temp workaround would be applying the unreleased fix in https://databricks.atlassian.net/browse/ES-1203418 directly to the SDK code in your laptop: $HOME/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/sdk/credentials_provider.py

sagpra commented 2 weeks ago

OK Thanks, I will check it again

JCZuurmond commented 1 week ago

Should be resolved in latest sdk release: https://github.com/databricks/databricks-sdk-py/releases/tag/v0.31.1
Dependabot will bump the dependency version in ucx shortly

JCZuurmond commented 6 days ago

Should be resolved when upgrading to the latest version databricks labs upgrade ucx. Make sure that you have the latest version of the Databricks cli before upgrading ucx. Please report back if the issue is not resolved.

databrickslabs / ucx