search

dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.37k stars 1.56k forks source link

[Bug] Type object 'MacroFuzzEnvironment' has no attribute 'concat' #9896

Closed endlessdb closed 5 days ago

endlessdb commented 3 months ago

Is this a new bug in dbt-core?

Current Behavior

Can someone take a look into this? I'm getting this error when I ran dbt command (compile or run) on the new version v.1.8.02b. , I don't see issue on 1.7.11

`dbt compile'

20:55:32 [WARNING]: Deprecated functionality User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml. 20:55:33 Running with dbt=1.8.0-b2 20:55:33 Registered adapter: snowflake=1.8.0-b2 20:55:33 Unable to do partial parsing because saved manifest not found. Starting full parse. 20:55:36 Encountered an error: type object 'MacroFuzzEnvironment' has no attribute 'concat' 20:55:36 Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 106, in wrapper result, success = func(*args, kwargs) File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 91, in wrapper return func(*args, *kwargs) File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 184, in wrapper return func(args, kwargs) File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 213, in wrapper return func(*args, *kwargs) File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 260, in wrapper return func(args, **kwargs) File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 286, in wrapper ctx.obj["manifest"] = parse_manifest( File "/usr/local/lib/python3.9/site-packages/dbt/parser/manifest.py", line 1854, in parse_manifest manifest = ManifestLoader.get_full_manifest( File "/usr/local/lib/python3.9/site-packages/dbt/parser/manifest.py", line 325, in get_full_manifest manifest = loader.load() File "/usr/local/lib/python3.9/site-packages/dbt/parser/manifest.py", line 430, in load self.parse_project( File "/usr/local/lib/python3.9/site-packages/dbt/parser/manifest.py", line 689, in parse_project parser.parse_file(block) File "/usr/local/lib/python3.9/site-packages/dbt/parser/base.py", line 484, in parse_file self.parse_node(file_block) File "/usr/local/lib/python3.9/site-packages/dbt/parser/base.py", line 445, in parse_node self.render_update(node, config) File "/usr/local/lib/python3.9/site-packages/dbt/parser/models.py", line 348, in render_update super().render_update(node, config) File "/usr/local/lib/python3.9/site-packages/dbt/parser/base.py", line 421, in render_update context = self.render_with_context(node, config) File "/usr/local/lib/python3.9/site-packages/dbt/parser/base.py", line 270, in render_with_context get_rendered(parsed_node.raw_code, context, parsed_node, capture_macros=True) File "/usr/local/lib/python3.9/site-packages/dbt/clients/jinja.py", line 146, in get_rendered rendered = render_template(template, ctx, node) File "/usr/local/lib/python3.9/site-packages/dbt_common/clients/jinja.py", line 539, in render_template return template.render(ctx) File "/usr/local/lib/python3.9/site-packages/dbt_common/clients/jinja.py", line 144, in render return self.environment.handle_exception() File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 925, in handle_exception raise rewrite_traceback_stack(source=source) AttributeError: type object 'MacroFuzzEnvironment' has no attribute 'concat'`

Expected Behavior

able to finish dbt command

Steps To Reproduce

dbt compile

Relevant log output

No response

Environment

- OS:Linux 3.10.0-1160.114.el7.x86_64 x86_64
- Python: 3.9
- dbt: 1.8.02b

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

dbeatty10 commented 3 months ago

We'd be happy to take a look @endlessdb. We've used that version a bit and haven't seen that issue pop up yet.

We'd just need a simple set of project files that reproduce the issue so we can get to the bottom of it. Could you provide that?

endlessdb commented 3 months ago

HI @dbeatty10 what kind of files do you needed and how do I share files with you?

dbeatty10 commented 3 months ago

Something like this:

### Reprex Create these files: `models/my_model.sql` ```sql select 1 as id, null as status ``` `models/_models.yml` ```yaml models: - name: my_model columns: - name: status tests: - not_null: name: 'Expect "node_id" values to be unique.' config: store_failures_as: view ``` Run this command: ```shell dbt build -s my_model ```

Here are a handful of examples:

olivewood1 commented 2 weeks ago

@dbeatty10 I have a same issue after upgraded 1.8 and causing issues from any of dbt commands( run, seed, compile and ....), and figured out that MacroFuzzEnvironment class was available in the 1.7.X, but not 1.8.X. https://github.com/dbt-labs/dbt-core/blob/v1.7.17/core/dbt/clients/jinja.py Is there any reason that you removed this class?

dbeatty10 commented 2 weeks ago

Thanks for following up on this @olivewood1 !

We moved MacroFuzzEnvironment to dbt-common within https://github.com/dbt-labs/dbt-core/pull/8906. You can read about the background in 1 2 3.

How are you installing and running dbt? I'm guessing there's something very specific to your environment or the way you are using dbt that is causing the error you are seeing.

olivewood1 commented 2 weeks ago

Thanks @dbeatty10 . We built docker image with dbt_core and dbt-snowfalke , and run dbt commands, never used dbt-common. Do we need to include dbt-common to the docker container?

olivewood1 commented 1 week ago

@dbeatty10 is it required jinja2 version 3.1.3 or higher?

dbt-common 1.4.0 requires jinja2<4,>=3.1.3, but you have jinja2 3.0.3 which is incompatible. dbt-core 1.8.2 requires Jinja2<4,>=3.1.3, but you have jinja2 3.0.3 which is incompatible.

also MacrosFuzzEnvironment is something different between 1.7 and 1.8

dbeatty10 commented 1 week ago

@olivewood1 Which version of dbt are you expecting to install? And which command(s) are you using?

For dbt v1.7 and earlier, installing/upgrading dbt can look something like this:

pip install --upgrade "dbt-core~=1.7" "dbt-bigquery~=1.7"

For dbt v1.8 and later, it might be something like this:

pip install --upgrade "dbt-core~=1.8" dbt-common dbt-adapters dbt-bigquery

dbt's current requirement for Jinja2 is >=3.1.3,<4 which can be seen here and here.

olivewood1 commented 1 week ago

Looks like issue was resolved after upgrading the Jinja2 version to 3.1.2. Previously, it was running with version 3.0.3 in the container. Thank you so much @dbeatty10