elementary-data / elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
https://www.elementary-data.com/
Apache License 2.0
1.87k stars 158 forks source link

Install elementary dbt package breaks dbt #1628

Open opiero opened 2 months ago

opiero commented 2 months ago

Describe the bug After running dbt deps to install elementary data, I am unable to run dbt run -s elementary or many other dbt commands such as run, run-operation, compile and test.

Whenever one of these commands runs, the following error is shown:

18:03:38  Encountered an error:
Compilation Error in operation elementary-on-run-start-0 (./dbt_project.yml)
  'None' has no attribute 'database'

To Reproduce Steps to reproduce the behavior:

  1. Add elementary-data/elementary to your packages.yml
  2. Add elementary to your dbt_project.yml as described in the official documentation
  3. Run dbt deps
  4. Run any of the above cited dbt commands, such as dbt run --select elementary

Expected behavior A clear and concise description of what you expected to happen. I expected dbt to run normally without errors.

Screenshots If applicable, add screenshots to help explain your problem. Screenshot from 2024-07-08 15-16-00

Environment (please complete the following information):

Additional context dbt debug says that all checks passed.

Would you be willing to contribute a fix for this issue? Yes

NoyaArie commented 1 month ago

I could not reproduce this error. Can you please share the dbt.log as well?

opiero commented 1 month ago

Of course! There you go:

16:13:28.712021 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f04c3b1cad0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f04c3c0f590>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f04c3c25cd0>]}

============================== 16:13:28.714661 | 18bf1616-5642-45ff-85cf-8310951f7e03 ==============================
16:13:28.714661 [info ] [MainThread]: Running with dbt=1.7.14
16:13:28.715213 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'fail_fast': 'False', 'version_check': 'True', 'log_path': 'projects/dev/logs', 'debug': 'False', 'profiles_dir': 'profiles', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'log_format': 'default', 'introspect': 'True', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'target_path': 'None', 'invocation_command': 'dbt run --profiles-dir profiles --project-dir projects/dev --select elementary', 'send_anonymous_usage_stats': 'True'}
16:13:29.415063 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '18bf1616-5642-45ff-85cf-8310951f7e03', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f04975dfd70>]}
16:13:29.463531 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '18bf1616-5642-45ff-85cf-8310951f7e03', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0497f30320>]}
16:13:29.464329 [info ] [MainThread]: Registered adapter: bigquery=1.7.7
16:13:29.482790 [debug] [MainThread]: checksum: ccbe0bccecb23e0b0bcca76d525b7d642e28acd474046360f7f33e61c6a2b781, vars: {}, profile: , target: , version: 1.7.14
16:13:29.530521 [info ] [MainThread]: Unable to do partial parsing because config vars, config profile, or config target have changed
16:13:29.531096 [debug] [MainThread]: previous checksum: ccbe0bccecb23e0b0bcca76d525b7d642e28acd474046360f7f33e61c6a2b781, current checksum: 5b6220cc0ac20684143ed8202a0eacaa9022d0d316a150c1755cdaea14655d9f
16:13:29.531441 [info ] [MainThread]: Unable to do partial parsing because a project dependency has been added
16:13:29.531777 [info ] [MainThread]: Unable to do partial parsing because a project config has changed
16:13:29.532140 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': '18bf1616-5642-45ff-85cf-8310951f7e03', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0496adbe60>]}
16:13:35.714620 [error] [MainThread]: Encountered an error:
Compilation Error in operation elementary-on-run-start-0 (./dbt_project.yml)
  'None' has no attribute 'database'
16:13:35.715641 [debug] [MainThread]: Resource report: {"command_name": "run", "command_wall_clock_time": 7.0393953, "process_user_time": 8.266332, "process_kernel_time": 0.925436, "process_mem_max_rss": "217460", "process_out_blocks": "16", "command_success": false, "process_in_blocks": "0"}
16:13:35.716306 [debug] [MainThread]: Command `dbt run` failed at 16:13:35.716156 after 7.04 seconds
16:13:35.716739 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f04c402a480>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0496759fd0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f04968f4650>]}
16:13:35.717150 [debug] [MainThread]: Flushing usage events
NoyaArie commented 1 month ago

You added Elementary to the packages.json file, and modified dbt_project.yml, and nothing else, right?

In your dbt project, did you set an on-run-start logic or override some macros?

opiero commented 1 month ago

I added Elementary to packages.yml. Does that make a difference? I have dbt_utils and dbt_date working fine that way. Aside from that, we overrode generate_schema_name macro, could it be that?