search

z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
419 stars 46 forks source link

Bug: dbt-osmosis 0.13.1 is not compatible with dbt 1.8.0+ #161

Closed FrankTub closed 1 week ago

FrankTub commented 1 week ago

Run ddbt-osmosis yaml document in an environment with dbt-osmosis 0.13.1, dbt-core 1.8.2 and dbt-postgres 1.8.1 and the resulting error is:

INFO     🌊 Executing dbt-osmosis                                                                                                                                                                                        main.py:511

13:21:12  [WARNING]: Deprecated functionality
The `tests` config has been renamed to `data_tests`. Please see
https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax for more information.
Traceback (most recent call last):
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/dbt_osmosis/main.py", line 80, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/dbt_osmosis/main.py", line 513, in document
    runner = DbtYamlManager(
             ^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/dbt_osmosis/core/osmosis.py", line 130, in __init__
    super().__init__(target, profiles_dir, project_dir, threads, vars=vars, profile=profile)
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 391, in __init__
    self.parse_project(init=True)
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 480, in parse_project
    self.manifest = _project_parser.load()
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/dbt/parser/manifest.py", line 528, in load
    self.check_for_spaces_in_resource_names()
  File "/Users/frank.tubbing/source/repos/data-warehouse-etl-dbt/venv/lib/python3.11/site-packages/dbt/parser/manifest.py", line 641, in check_for_spaces_in_resource_names
    if self.root_project.args.REQUIRE_RESOURCE_NAMES_WITHOUT_SPACES
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DbtConfiguration' object has no attribute 'REQUIRE_RESOURCE_NAMES_WITHOUT_SPACES'
shinyafw commented 1 week ago

I'm experiencing the same issue with dbt-osmosis 0.13.1, dbt-core 1.8.2 and dbt-snowflake 1.8.3.

I was able to resolve the issue by adding the REQUIRE_RESOURCE_NAMES_WITHOUT_SPACES field to the DbtConfiguration class and by passing some contexts (See https://github.com/shinyafw/dbt-osmosis/commit/26e0b8434ee66b3b2cc3ee2fb6a77cefdc338459). But I'm not certain that it's the proper fix.

barthichop commented 1 week ago

We're experiencing the same issue with dbt-osmosis 0.13.1, dbt-core 1.8.0 and dbt-bigquery 1.8.0

kokorin commented 1 week ago

I also have that issue, and created a PR #160

illia-yurtsiv-proxet commented 1 week ago

Getting this with dbt-osmosis 0.13.1, dbt-core 1.8.2 and dbt-bigquery 1.8.1, running dbt-osmosis yaml refactor <filename>.sql

INFO     🌊 Executing dbt-osmosis                                                                                                                         main.py:219

Traceback (most recent call last):
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_osmosis/main.py", line 80, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_osmosis/main.py", line 221, in refactor
    runner = DbtYamlManager(
             ^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_osmosis/core/osmosis.py", line 130, in __init__
    super().__init__(target, profiles_dir, project_dir, threads, vars=vars, profile=profile)
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 391, in __init__
    self.parse_project(init=True)
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_osmosis/vendored/dbt_core_interface/project.py", line 471, in parse_project
    self.config = RuntimeConfig.from_args(self.base_config)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/runtime.py", line 275, in from_args
    project, profile = cls.collect_parts(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/runtime.py", line 254, in collect_parts
    profile = cls.get_profile(
              ^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/runtime.py", line 107, in get_profile
    return load_profile(
           ^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/runtime.py", line 71, in load_profile
    profile = Profile.render(
              ^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/profile.py", line 403, in render
    return cls.from_raw_profiles(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/profile.py", line 369, in from_raw_profiles
    return cls.from_raw_profile_info(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/profile.py", line 315, in from_raw_profile_info
    target_name, profile_data = cls.render_profile(
                                ^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/profile.py", line 284, in render_profile
    profile_data = renderer.render_data(raw_profile_data)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/renderer.py", line 53, in render_data
    return deep_map_render(self.render_entry, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_common/utils/dict.py", line 129, in deep_map_render
    return _deep_map_render(func, value, ())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_common/utils/dict.py", line 94, in _deep_map_render
    ret = {k: _deep_map_render(func, v, (keypath + (str(k),))) for k, v in value.items()}
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_common/utils/dict.py", line 94, in <dictcomp>
    ret = {k: _deep_map_render(func, v, (keypath + (str(k),))) for k, v in value.items()}
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_common/utils/dict.py", line 96, in _deep_map_render
    ret = func(value, keypath)
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/renderer.py", line 37, in render_entry
    return self.render_value(value, keypath)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/renderer.py", line 203, in render_value
    raise ex
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/renderer.py", line 196, in render_value
    rendered = super().render_value(value, keypath)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/config/renderer.py", line 46, in render_value
    return get_rendered(value, self.context, native=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/clients/jinja.py", line 146, in get_rendered
    rendered = render_template(template, ctx, node)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 539, in render_template
    return template.render(ctx)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 228, in render
    return self.environment.handle_exception()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 184, in quoted_native_concat
    head = list(islice(nodes, 2))
           ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 1, in top-level template code
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt/context/secret.py", line 32, in env_var
    env = get_invocation_context().env
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/illiayurtsiv/dev-projects/proxet/Railsreactor/gc-proxet-poc-dbt/.venv/lib/python3.11/site-packages/dbt_common/context.py", line 53, in get_invocation_context
    ctx = invocation_var.get()
          ^^^^^^^^^^^^^^^^^^^^
LookupError: <ContextVar name='DBT_INVOCATION_CONTEXT_VAR' at 0x10744c4a0>
kokorin commented 1 week ago

@illia-yurtsiv-proxet It's fixed in #160 but not yet released. I believe #164 is also required to support DBT 1.8

illia-yurtsiv-proxet commented 1 week ago

@kokorin My issue is slightly different though. Made a fix in #165.

z3z1ma commented 1 week ago

Thanks for the contributions @kokorin & @illia-yurtsiv-proxet -- especially the sanity tests @kokorin which were a long time coming and should make it much more straightforward to identify any obvious misses.

These have been merged and a new release cut to PyPi, 0.13.2