search

z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
456 stars 46 forks source link

'DbtYamlManager' object has no attribute 'yaml', No module named 'dbt_osmosis' #17

Closed coryandrewtaylor closed 1 year ago

coryandrewtaylor commented 1 year ago

Hi there--thanks for the great tool!

I'm getting an AttributeError when running dbt-osmosis compose and dbt-osmosis run. See the stacktrace below. My dbt_project.yml is attached, as well.

> dbt-osmosis run -f acdw.dimensions.acdw__dimensions__item 
INFO     🌊 Executing dbt-osmosis                                                                                                                                                                                                                                         main.py:78

INFO     📈 Searching project stucture for required updates and building action plan                                                                                                                                                                                  osmosis.py:907
INFO     ...building project structure mapping in memory                                                                                                                                                                                                              osmosis.py:885
INFO     [('CREATE', '->', WindowsPath('C:/my_project_path/models/acdw/dimensions/schema/acdw__dimensions__item.yml'))]                                                                                                                         osmosis.py:1027
INFO     👷 Executing action plan and conforming projecting schemas to defined structure                                                                                                                                                                              osmosis.py:977
INFO     🚧 Building schema file acdw__dimensions__item.yml                                                                                                                                                                                                           osmosis.py:983
Traceback (most recent call last):
  File "C:\Users\me\Anaconda3\envs\dbt\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\me\Anaconda3\envs\dbt\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\me\Anaconda3\envs\dbt\Scripts\dbt-osmosis.exe\__main__.py", line 7, in <module>
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt_osmosis\main.py", line 89, in run
    if runner.commit_project_restructure_to_disk():
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt_osmosis\core\osmosis.py", line 987, in commit_project_restructure_to_disk
    self.yaml.dump(structure.output, target)
AttributeError: 'DbtYamlManager' object has no attribute 'yaml'

Possibly relatedly, I'm not able to run dbt-osmosis workbench, either. Apparently, dbt_osmosis can't find itself.

> dbt-osmosis workbench
INFO     🌊 Executing dbt-osmosis                                                                                                                                                                                                                                        main.py:406
[...]
2022-10-06 17:41:59.975 Pandas backend loaded 1.4.2
2022-10-06 17:41:59.981 Numpy backend loaded 1.21.5
2022-10-06 17:41:59.982 Pyspark backend NOT loaded
2022-10-06 17:41:59.983 Python backend loaded
2022-10-06 17:42:01.020 Uncaught app exception
Traceback (most recent call last):
  File "C:\Users\me\Anaconda3\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 562, in _run_script
    exec(code, module.__dict__)
  File "C:\Users\me\Anaconda3\envs\dbt\Lib\site-packages\dbt_osmosis\app.py", line 17, in <module>
    from dbt_osmosis.core.osmosis import DEFAULT_PROFILES_DIR, DbtProject
ModuleNotFoundError: No module named 'dbt_osmosis'

dbt_project.zip

z3z1ma commented 1 year ago

@coryandrewtaylor

Thanks for the issue!

So the DbtYamlManager was using an old accessor so this has been corrected in 0.8.7

The workbench is another issue 🤔 You installed it with the extra I presume dbt-osmosis[workbench] ?

Let me know if it works after bumping the version

coryandrewtaylor commented 1 year ago

@z3z1ma Thanks for the super fast reply!

I've upgraded to v0.8.7 and installed the workbench extra. commit_project_restructure_to_disk is still erroring out, though now it's a TypeError:

Traceback (most recent call last):
  File "C:\Users\me\Anaconda3\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\me\Anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\me\.local\bin\dbt-osmosis.exe\__main__.py", line 7, in <module>
  File "C:\Users\me\.local\pipx\venvs\dbt-osmosis\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\me\.local\pipx\venvs\dbt-osmosis\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\me\.local\pipx\venvs\dbt-osmosis\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\me\.local\pipx\venvs\dbt-osmosis\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\me\.local\pipx\venvs\dbt-osmosis\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\me\.local\pipx\venvs\dbt-osmosis\lib\site-packages\dbt_osmosis\main.py", line 89, in run
    if runner.commit_project_restructure_to_disk():
  File "C:\Users\me\.local\pipx\venvs\dbt-osmosis\lib\site-packages\dbt_osmosis\core\osmosis.py", line 997, in commit_project_restructure_to_disk
    if "version" not in target_schema:
TypeError: argument of type 'NoneType' is not iterable

The full log is here: dbt_osmosis_run.log

I'm able to load the workbench now, though only in a conda environment; I get the same ModuleNotFoundError in a pipx venv. Local SQL compilation works just fine. However, when trying to test the compiled query, I get dbt.exceptions.InvalidConnectionException: Runtime Error: connection never acquired for thread (29328, 3368), have [(29328, 29948)]. The traceback and full logs are below.

Traceback (most recent call last):
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 558, in _run_script
    self._session_state.on_script_will_rerun(rerun_data.widget_states)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\streamlit\runtime\state\safe_session_state.py", line 73, in on_script_will_rerun
    self._state.on_script_will_rerun(latest_widget_states)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\streamlit\runtime\state\session_state.py", line 543, in on_script_will_rerun
    self._call_callbacks()
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\streamlit\runtime\state\session_state.py", line 556, in _call_callbacks
    self._new_widget_state.call_callback(wid)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\streamlit\runtime\state\session_state.py", line 278, in call_callback
    callback(*args, **kwargs)
  File "C:\Users\me\Anaconda3\envs\dbt\Lib\site-packages\dbt_osmosis\app.py", line 205, in run_query
    result = ctx.execute_sql(f"select * from ({sql}) as __all_data limit {limit}")
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt_osmosis\core\osmosis.py", line 440, in execute_sql
    *self.adapter_execute(compiled_sql, fetch=True),
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt_osmosis\core\osmosis.py", line 422, in adapter_execute
    return self.adapter.execute(sql, auto_begin, fetch)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt\adapters\base\impl.py", line 242, in execute
    return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt\adapters\snowflake\connections.py", line 441, in execute
    _, cursor = self.add_query(sql, auto_begin)
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt\adapters\snowflake\connections.py", line 492, in add_query
    connection, cursor = super().add_query(
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt\adapters\sql\connections.py", line 55, in add_query
    connection = self.get_thread_connection()
  File "C:\Users\me\Anaconda3\envs\dbt\lib\site-packages\dbt\adapters\base\connections.py", line 91, in get_thread_connection
    raise dbt.exceptions.InvalidConnectionException(key, list(self.thread_connections))
dbt.exceptions.InvalidConnectionException: Runtime Error
  connection never acquired for thread (29328, 3368), have [(29328, 29948)]

dbt_osmosis_workbench.log

As a side note--it seems like you've caught a bug in Streamlit! See lines 14-168 in the log for that error.

coryandrewtaylor commented 1 year ago

@z3z1ma Doing a bit of debugging, the NoneType is not iterable error was coming from DbtYamlManager.yaml_handler reading an empty yaml file. Adding in a couple of lines to create a new dict in that case resolves the problem.

I've put in PR #18 with the fix.

z3z1ma commented 1 year ago

Thanks @coryandrewtaylor for that fix!

Also super interesting on the connection piece. I think I know roote cause and can address that too. I have been working with connections in general as well as snowflake stuff and have my head wrapped around whats needed.

z3z1ma commented 1 year ago

The last set of changes should have resolved connection issues. We can reopen another specific issue if it resurfaces.