search

elementary-data / elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
https://www.elementary-data.com/
Apache License 2.0
1.94k stars 165 forks source link

Report fails to generate if exposure owners is string #1732

Open arthurcw opened 3 weeks ago

arthurcw commented 3 weeks ago

Describe the bug Could not generate the report - Error: validation error for ExposureSchema. It's similar to #827

To Reproduce Configure the exposure owner to:

owner:
  name: abcdefg

Expected behavior edr report or edr send-report to run without errors

Screenshots

INFO — Running with edr=0.16.1
INFO — Installing packages for edr internal dbt package...
INFO — Running dbt command --log-format json deps --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
INFO — edr (0.16.1) and Elementary's dbt package (0.16.0) are compatible.
INFO — Elementary's database and schema: '"abc.efg"'
INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
 INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
ERROR — Could not generate the report - Error: 1 validation error for ExposureSchema
INFO - [base] owners
INFO - [base]   value is not a valid list (type=type_error.list)
INFO - [base] Please reach out to our community for help with this issue.
INFO - [base] Traceback (most recent call last):
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/elementary/monitor/api/report/report.py", line 72, in get_report_data
INFO - [base]     exposures = models_api.get_exposures(upstream_node_ids=lineage_node_ids)
INFO - [base]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/elementary/monitor/api/models/models.py", line 158, in get_exposures
INFO - [base]     exposures_results = self.models_fetcher.get_exposures()
INFO - [base]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/elementary/monitor/fetchers/models/models.py", line 72, in get_exposures
INFO - [base]     exposures = [ExposureSchema(**exposure) for exposure in exposures]
INFO - [base]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/elementary/monitor/fetchers/models/models.py", line 72, in <listcomp>
INFO - [base]     exposures = [ExposureSchema(**exposure) for exposure in exposures]
INFO - [base]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
INFO - [base]     raise validation_error
INFO - [base] pydantic.v1.error_wrappers.ValidationError: 1 validation error for ExposureSchema
INFO - [base] owners
INFO - [base]   value is not a valid list (type=type_error.list)

Environment (please complete the following information):

Additional context I think we need updates to the _load_var_to_list method here

  1. At this step, expsoure["owners"] = "abcdefg"
  2. When running load_owners method in ArtifactSchema, it calls the cls._load_var_to_list(owners)method
  3. _load_var_to_list will return abcdefg and not a list

Would you be willing to contribute a fix for this issue? Yes. I think either need to adjust how the owners is returned from self.models_fetcher.get_exposures() or adjust __load_var_to_list method.

haritamar commented 3 weeks ago

Thanks for reporting @arthurcw ! Interesting - I'd expect _load_var_to_list to work correctly since try_load_json would return None and we'd return a list with a single string as a result.

Though admittedly the code is confusing so maybe there's a bug there. In any case if you have a fix please feel free to submit a PR.

Thanks! Itamar