Error: `TypeError: cannot pickle '_thread.RLock' object`

[jaredx435k2d0]

Loving how much this project is charging ahead!

Was forced to be away from it for a while due to some fires to put out. Anxious to explore improvements and new features.

One thing I've always encountered (including on version 0.11.14) is getting: TypeError: cannot pickle '_thread.RLock' object randomly when trying to execute yaml refactor.

Traceback:

dbt-osmosis yaml refactor --fqn staging.[MY_FOLDER_NAME_REDACTED]
INFO     🌊 Executing dbt-osmosis                                                                                                                                 main.py:141

Traceback (most recent call last):
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt_osmosis/main.py", line 81, in wrapper
    return func(*args, **kwargs)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt_osmosis/main.py", line 143, in refactor
    runner = DbtYamlManager(
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt_osmosis/core/osmosis.py", line 119, in __init__
    super().__init__(  # partial parse messes up our f strings, so force a full parse on init
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 409, in __init__
    self.parse_project(init=True)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 493, in parse_project
    self.manifest = _project_parser.load()
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt/parser/manifest.py", line 351, in load
    self.parse_project(
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt/parser/manifest.py", line 479, in parse_project
    parser.parse_file(block)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt/parser/base.py", line 414, in parse_file
    self.parse_node(file_block)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt/parser/base.py", line 388, in parse_node
    self.render_update(node, config)
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt/parser/models.py", line 306, in render_update
    model_parser_copy = self.partial_deepcopy()
  File "[MY_FILE_PATH_REDACTED]/dbt_oc__venv/lib/python3.10/site-packages/dbt/parser/models.py", line 461, in partial_deepcopy
    return ModelParser(deepcopy(self.project), self.manifest, deepcopy(self.root_project))
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.RLock' object

I usually just have to keep trying to execute and eventually it'll work. Sometimes I'll quit my terminal app and force quit Python from the Activity Monitor and this seems to work pretty reliably.

[z3z1ma]

https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/parser/models.py#L304

Looks like some janky behavior related to whatever is going on in the static parser / experimental parser code. They are pickling objects (we don't need to pickle anything because we are supposed to be running in one process) and using random sampling to randomly run their correctness tests or something on all the dbt projects in the wild. As far as I can gather. 🤷

I can find a way to patch that out or force the appropriate flag so we dont have to deal with that

[z3z1ma]

Not really a fan of that kind of subtle sampling without it being better advertised. 🙅

[z3z1ma]

@jaredx435k2d0 This has been addressed in latest release 0.11.16 🎉