search

z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
456 stars 46 forks source link

[bug?]dbt-osmosis 0.13 Fails with dbt 1.8.2 "TypeError: BaseAdapter.__init__() missing 1 required positional argument: 'mp_context'" #159

Closed toukoudo closed 3 months ago

toukoudo commented 3 months ago

Hi,

(Just a guess. Sorry if gibberish) I think dbt-osmosis uses a vendored dbt core interface. Since adapters are decoupled in dbt 1.8, could the changes in signatures have caused this error?

error

> dbt-osmosis yaml refactor someawesomemodel

INFO     🌊 Executing dbt-osmosis                                                                                                                                                                          main.py:219

02:34:11  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
02:34:13  [WARNING]: Deprecated functionality
The `tests` config has been renamed to `data_tests`. Please see
https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax for more
information.
Traceback (most recent call last):
  File "/Users/makotokimura/project/dataops-dbt/.venv/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/main.py", line 80, in wrapper
    return func(*args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/main.py", line 221, in refactor
    runner = DbtYamlManager(
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/core/osmosis.py", line 130, in __init__
    super().__init__(target, profiles_dir, project_dir, threads, vars=vars, profile=profile)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 392, in __init__
    self.parse_project(init=True)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 468, in parse_project
    self.initialize_adapter()
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 424, in initialize_adapter
    self.adapter = self.get_adapter_cls()(self.config)
TypeError: BaseAdapter.__init__() missing 1 required positional argument: 'mp_context'

my environment

> dbt-osmosis --version
dbt-osmosis, version 0.13.0

> dbt --version
Core:
  - installed: 1.8.2
  - latest:    1.8.2 - Up to date!

Plugins:
  - snowflake: 1.8.3 - Up to date!

> pip freeze | grep dbt
dagster-dbt==0.23.8
dbt-adapters==1.2.1
dbt-common==1.3.0
dbt-core==1.8.2
dbt-extractor==0.5.1
dbt-metricflow==0.6.0
dbt-osmosis==0.13.0
dbt-postgres==1.4.5
dbt-redshift==1.4.0
dbt-semantic-interfaces==0.5.1
dbt-snowflake==1.8.3
sqlfluff-templater-dbt==3.0.7
z3z1ma commented 3 months ago

Thanks for pointing this out. I released a patch fix, 0.13.1 that should solve for this. Please let me know if this works for you.

toukoudo commented 3 months ago

Thank you for the quick fix!

Please let me know if this works for you.

Well, actually no.

What I have done:

  1. updated osmosis
  2. reinstalled the adapter package
    • Without reinstalling, osmosis causes an error 'ModuleNotFoundError: No module named 'dbt.adapters.base.relation'"
  3. ran the "dbt-osmosis yaml refactor" command, which resulted in the following error

version

 pip freeze | grep dbt
dagster-dbt==0.23.8
dbt-adapters==1.2.1
dbt-common==1.3.0
dbt-core==1.8.2
dbt-extractor==0.5.1
dbt-metricflow==0.6.0
dbt-osmosis==0.13.1
dbt-postgres==1.4.5
dbt-redshift==1.4.0
dbt-semantic-interfaces==0.5.1
dbt-snowflake==1.8.3
sqlfluff-templater-dbt==3.0.7

ran osmosis command:

dbt-osmosis yaml refactor someawesomemodel.sql
INFO     🌊 Executing dbt-osmosis                                                                                                                                                                          main.py:219

10:14:56  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
10:14:58  [WARNING]: Deprecated functionality
The `tests` config has been renamed to `data_tests`. Please see
https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax for more
information.
Traceback (most recent call last):
  File "/Users/makotokimura/project/dataops-dbt/.venv/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/main.py", line 80, in wrapper
    return func(*args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/main.py", line 221, in refactor
    runner = DbtYamlManager(
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/core/osmosis.py", line 130, in __init__
    super().__init__(target, profiles_dir, project_dir, threads, vars=vars, profile=profile)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 391, in __init__
    self.parse_project(init=True)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 480, in parse_project
    self.manifest = _project_parser.load()
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/parser/manifest.py", line 435, in load
    self.parse_project(
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/parser/manifest.py", line 733, in parse_project
    parser.parse_file(block)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/parser/base.py", line 484, in parse_file
    self.parse_node(file_block)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/parser/base.py", line 445, in parse_node
    self.render_update(node, config)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/parser/models.py", line 252, in render_update
    super().render_update(node, config)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/parser/base.py", line 421, in render_update
    context = self.render_with_context(node, config)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/parser/base.py", line 270, in render_with_context
    get_rendered(parsed_node.raw_code, context, parsed_node, capture_macros=True)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/clients/jinja.py", line 146, in get_rendered
    rendered = render_template(template, ctx, node)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_common/clients/jinja.py", line 539, in render_template
    return template.render(ctx)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_common/clients/jinja.py", line 144, in render
    return self.environment.handle_exception()
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 11, in top-level template code
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/clients/jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_common/clients/jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 2, in template
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt/context/providers.py", line 1388, in env_var
    env = get_invocation_context().env
  File "/Users/makotokimura/project/dataops-dbt/.venv/lib/python3.10/site-packages/dbt_common/context.py", line 53, in get_invocation_context
    ctx = invocation_var.get()
LookupError: <ContextVar name='DBT_INVOCATION_CONTEXT_VAR' at 0x104b70180>