Upgrading version 0.15.1 → 0.16.0 cause "Runtime Error: Could not find profile named 'elementary"

phalanx-hk commented 2 months ago

Describe the bug Yesterday I upgrading elementary-data[bigquery] from 0.15.1 to 0.16.0, but Upgrading caused a Runtime Error. I include below the contents of the files and logs that are considered necessary for troubleshooting.

To Reproduce Steps to reproduce the behavior:

setting dependencies and relevant yaml file as below pyproject.toml

"dbt-core >=1.8,<1.9",
"dbt-bigquery >=1.8,<1.9",
"elementary-data[bigquery] ==0.16.0",

profiles.yml

elementary:
  target: prod
  outputs:
    prod:
      dataset: dbt_elementary
      job_execution_timeout_seconds: 300
      job_retries: 1
      location: asia-northeast1
      method: oauth
      priority: interactive
      project: {project_id}
      threads: 4
      type: bigquery

packages.yml

packages:
  - package: elementary-data/elementary
    version: 0.16.0

dbt_project.yml

models:
  ...
  elementary:
  +schema: "dbt_elementary"
  +docs:
    show: false
flags:
  # Required from dbt 1.8 and above for certain Elementary features
  require_explicit_package_overrides_for_builtin_materializations: false

run command

edr report --env prod --project-name {project_id}  --days-back=30

see error


________                          __                  
/ ____/ /__  ____ ___  ___  ____  / /_____ ________  __
/ __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
/ /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                             /____/

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-10 23:39:29 — INFO — Running with edr=0.16.0 2024-09-10 23:39:29 — INFO — Installing packages for edr internal dbt package... 2024-09-10 23:39:29 — INFO — Running dbt command --log-format json deps --project-dir {WORKING_DIR}/dbt/.venv/lib/python3.12/site-packages/elementary/monitor/dbt_project {"data": {"exc": "Runtime Error\n Could not find profile named 'elementary'"}, "info": {"category": "", "code": "Z002", "extra": {}, "invocation_id": "3ef899d9-d564-4170-a149-06641a6a5aac", "level": "error", "msg": "Encountered an error:\nRuntime Error\n Could not find profile named 'elementary'", "name": "MainEncounteredError", "pid": 33465, "thread": "MainThread", "ts": "2024-09-10T14:39:31.495433Z"}} 2024-09-10 23:39:32 — ERROR — Unable to get the latest invocation: Failed to run dbt command. Runtime Error Could not find profile named 'elementary'



**Expected behavior**
A report of the dbt project will be generated.

**Environment (please complete the following information):**

- Elementary CLI (edr) version: 0.16.0
- Elementary dbt package version: 0.16.0
- dbt version you're using: 1.8.6
- Data warehouse: bigquery

geo909 commented 2 months ago

I have the exact same issue after upgrading from 0.15.1 to 0.16.1.

> dbt --version                                                                                                                                                         BI-811-dbt-sources-Fix-freshness-tests-and-alerting [3059a3f4] modified
Core:
  - installed: 1.8.6
  - latest:    1.8.6 - Up to date!

Plugins:
  - bigquery: 1.8.2 - Up to date!

Note that I also upgraded dbt from 1.7.x (dbt-core was 1.7.10 and dbt-bigquery was 1.7.6) to 1.8 at the same time.

geo909 commented 2 months ago

Another tip, in case it helps.

I'm fixing the following:

# requirements.txt
dbt-core==1.8.6
dbt-bigquery==1.8.2

and

# packages.yml
packages
  - package: elementary-data/elementary
      version: 0.16.1

With python package elementary-data==0.16.1 (same problem with 0.16.0):


> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                       main [0e491059] modified
________                          __                  
/ ____/ /__  ____ ___  ___  ____  / /_____ ________  __
/ __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
/ /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                             /____/

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-17 10:19:10 — INFO — Running with edr=0.16.1 {"data": {"exc": "Database Error\n [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "c48be6d8-6481-490d-890d-0baec2937684", "level": "error", "msg": "Encountered an error while running operation: Database Error\n [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 24886, "thread": "MainThread", "ts": "2024-09-17T07:19:11.602498Z"}} 2024-09-17 10:19:12 — ERROR — Unable to get the latest invocation: Failed to run dbt command. 2024-09-17 10:19:14 — INFO — Elementary's database and schema: '"bigquery-xxxx.user_gtzanakis_dbt"' 2024-09-17 10:19:14 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper {"data": {"exc": "Database Error\n [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "43151044-9715-4608-b6f1-a1fa68f06067", "level": "error", "msg": "Encountered an error while running operation: Database Error\n [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 24886, "thread": "MainThread", "ts": "2024-09-17T07:19:15.226170Z"}} 2024-09-17 10:19:15 — ERROR — Could not generate the report - Error: Failed to run dbt command. Please reach out to our community for help with this issue. Traceback (most recent call last): File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data tests_api = TestsAPI( File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in init self.test_results_db_rows = self._get_test_results_db_rows( File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows return self.tests_fetcher.get_all_test_results_db_rows( File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows run_operation_response = self.dbt_runner.run_operation( File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 177, in run_operation result = self._run_command( File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 114, in _run_command result = self._inner_run_command( File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command raise DbtCommandError( elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.


2. With python package `elementary-data==0.15.1`:

edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper" main [0e491059] modified

/ __/ / __ __ _ / /____ ____ / / / / _ \/ __ \/ _ \/ __ \/ __/ __/ / / / / / // / / / / / / / / / / / // // / / / // / /____/_/_// // //\// //_/_,// __, /
/____/

You are using Elementary 0.15.1, however version 0.16.1 is available. Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-17 10:24:21 — INFO — Running with edr=0.15.1 2024-09-17 10:24:21 — INFO — edr internal dbt package is not up-to-date, updating it... 2024-09-17 10:24:21 — INFO — Running dbt --log-format json deps --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:24:25 — INFO — Running with dbt=1.8.6 2024-09-17 10:24:25 — INFO — Updating lock file in file path: /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project/package-lock.yml 2024-09-17 10:24:25 — INFO — Installing dbt-labs/dbt_utils 2024-09-17 10:24:25 — INFO — Installed from version 0.8.6 2024-09-17 10:24:25 — INFO — Updated version available: 1.3.0 2024-09-17 10:24:25 — INFO — Installing elementary-data/elementary 2024-09-17 10:24:25 — INFO — Installed from version 0.15.1 2024-09-17 10:24:25 — INFO — Updated version available: 0.16.1 2024-09-17 10:24:25 — INFO — 2024-09-17 10:24:25 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']
Update your versions in packages.yml, then run dbt deps 2024-09-17 10:24:44 — INFO — edr (0.15.1) and Elementary's dbt package (0.15.2) are compatible. 2024-09-17 10:24:49 — INFO — Elementary's database and schema: '"bigquery-xxxx.user_gtzanakis_dbt"' 2024-09-17 10:24:49 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:24:57 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:04 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:11 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:18 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:25 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:31 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:38 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:45 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:51 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper 2024-09-17 10:25:58 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper



I don't know if it makes a difference but mentioning it just in case: I'm using pyenv virtualenv in ubuntu to handle my virtual environments, hopefully that's not the issue here.

ofek1weiss commented 2 months ago

Hey, @phalanx-hk , is your profiles.yml file located in your project directory or in the user directory?

if it is in the project directory, can you try adding --profiles-dir . to your command.

@geo909 from the logs you provided, i assume that in your profiles.yml file, i assume you define the following:

    keyfile: service-account-dbt.json

can you try setting the keyfile path to the absolute path of the file and see if this helps, something like

   keyfile: "/home/<user>/.../service-account-dbt.json

geo909 commented 2 months ago

@ofek1weiss Still doesn't work:

ferryhopper:
  target: dev_user
  outputs:
    dev_user:
      dataset: user_{{ env_var('DBT_USER') }}_dbt
      job_execution_timeout_seconds: 600
      job_retries: 1
      keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
      location: EU
      method: service-account
      priority: interactive
      project: bigquery-xxxxxx
      threads: 2
      type: bigquery

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                                                                                                                main [3a3a10e9] modified untracked
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-24 17:49:36 — INFO — Running with edr=0.16.0
2024-09-24 17:49:36 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path 'ferryhopper' does not exist.
2024-09-24 17:49:36 — ERROR — Failed to parse Elementary's database and schema.
2024-09-24 17:49:36 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-24 17:49:36 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Path 'ferryhopper' does not exist.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path 'ferryhopper' does not exist.

Tried this one too, just in case:

tzanakis@FHLT7FD7DL3:~/git/bi/dbt (*)
> edr report --project-dir "/home/tzanakis/git/bi/dbt/ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"                                                                                                                                                                                     main [3a3a10e9] untracked
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-24 17:53:23 — INFO — Running with edr=0.16.0
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 226007, "thread": "MainThread", "ts": "2024-09-24T14:53:25.257806Z"}}
2024-09-24 17:53:25 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
2024-09-24 17:53:28 — INFO — Elementary's database and schema: '"bigquery-xxxxxx.user_gtzanakis_dbt"'
2024-09-24 17:53:28 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 226007, "thread": "MainThread", "ts": "2024-09-24T14:53:28.741599Z"}}
2024-09-24 17:53:29 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.

Note that, if I change the virtual environment to switch back to 0.15.1 and change absolutely nothing else, both the above commands run without errors.

phalanx-hk commented 2 months ago

@ofek1weiss Thanks for your feedback! I try adding --profiles-dir ., it works for me.

run command: edr report --env prod --project-name {project_id} --days-back=30 --profiles-dir .

ofek1weiss commented 2 months ago

@geo909 did you try both at the same time? (setting both the keyfile and the command flags to absolute paths)? or was it one at a time?

the reason i am asking is that in the logs for the second command, you can see the log No such file or directory: 'service-account-dbt.json', i would have expected to see No such file or directory: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"

geo909 commented 2 months ago

@geo909 did you try both at the same time? (setting both the keyfile and the command flags to absolute paths)? or was it one at a time?

the reason i am asking is that in the logs for the second command, you can see the log No such file or directory: 'service-account-dbt.json', i would have expected to see No such file or directory: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"

Oops.. Apologies, that was my mistake. I was trying the full path in the wrong profile.

So, the following combination finally worked!

keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"

Played around with different expressions just in case, but relative paths expressed in different ways do not work, as expected probably. For example, the following:

keyfile: "./ferryhopper/service-account-dbt.json"
edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"

ofek1weiss commented 2 months ago

@geo909 Firstly, that is great to hear :) I'll try to figure out what change caused the keyfile issue, and see if it is reversible. Regarding --profiles-dir, it should work with relative paths, so doing the following should work:

keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper" can you please try that and let me know if it works/share logs if it does not? 🙏

geo909 commented 2 months ago

@ofek1weiss

No, the last combination does not work. To recap:

Works

keyfile: "/home/tzanakis/git/bi/dbt/ferryhopper/service-account-dbt.json"
edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"                                                                                                                                                        main [3a3a10e9] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-25 12:04:17 — INFO — Running with edr=0.16.0
2024-09-25 12:04:24 — INFO — edr (0.16.0) and Elementary's dbt package (0.16.1) are compatible.
2024-09-25 12:04:26 — INFO — Elementary's database and schema: '"bigquery-xxxxxx.user_gtzanakis_dbt"'
2024-09-25 12:04:26 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:32 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:35 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:39 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:42 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:46 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:49 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:54 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:58 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:05:01 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:05:04 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper

Does NOT work

keyfile: "/home/tzanakis/git/bi/dbt/ferryhopper/service-account-dbt.json"
edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"                                                                                                                                                                                main [3a3a10e9] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-25 12:06:18 — INFO — Running with edr=0.16.0
2024-09-25 12:06:18 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path './ferryhopper' does not exist.
2024-09-25 12:06:18 — ERROR — Failed to parse Elementary's database and schema.
2024-09-25 12:06:18 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ./ferryhopper
2024-09-25 12:06:18 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Path './ferryhopper' does not exist.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path './ferryhopper' does not exist.

Let me know if you need to me to try anything else!

ofek1weiss commented 2 months ago

ohhhhhhh, you use elementary 0.16.0, in 0.16.1 it should work (it is the only difference between them) @geo909

geo909 commented 1 month ago

ohhhhhhh, you use elementary 0.16.0, in 0.16.1 it should work (it is the only difference between them) @geo909

Ough.. I had tried with 0.16.1 earlier (see here) with the error again but I must have switched to 0.16.0 when playing around with the versions. Apologies again!

But we're getting there. Indeed now 0.16.1 works with the following combination:

edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"
keyfile: "/home/tzanakis/Tmp/bi.bak/dbt/ferryhopper/service-account-dbt.json"

But if I switch the keyfile parameter in profiles.yml to simply keyfile: service-account-dbt.json it is going to fail again.

Note that dbt runs fine with dbt run --project-dir "ferryhopper" --profiles-dir "ferryhopper" with the parameter set as keyfile: service-account-dbt.json.

geo909 commented 1 month ago

@ofek1weiss I cloned elementary's repo and played around, the issue is with this commit.

If you comment-out the line cwd=self.project_dir, my edr command works fine with keyfile: service-account-dbt.json, put it back and it breaks again.

Edit: Tried with the latest version of elementary, and it's the same if I comment out this line, this is what brakes it for me.

ofek1weiss commented 1 month ago

@geo909 i see, thanks for the deep dive 🙏 this line was added to overcome this issue in dbt, a fix for it has been merged, but they did not yet release the fixed version, once they do we will be able to change it

geo909 commented 1 month ago

@geo909 i see, thanks for the deep dive 🙏 this line was added to overcome this issue in dbt, a fix for it has been merged, but they did not yet release the fixed version, once they do we will be able to change it

Great, good to know! Do you have an estimate about how long this could take, by any chance?

elementary-data / elementary

Upgrading version 0.15.1 → 0.16.0 cause "Runtime Error: Could not find profile named 'elementary" #1702

Works

Does NOT work