greatexpectationslabs / ge_tutorials

Learn how to add data validation and documentation to a data pipeline built with dbt and Airflow.
166 stars 83 forks source link

execution error from task_transform_data_in_db #20

Closed sleepo-chin closed 3 years ago

sleepo-chin commented 3 years ago

I try to run docker-compose file for this tutorial but It error in this task. I don't know that what are you use dbt and greatexpectation version ? for this tutorial. Thank you very much

Reading local file: /usr/local/airflow/logs/ge_tutorials_dag_with_ge/task_transform_data_in_db/2021-05-17T17:02:07.041650+00:00/1.log
[2021-05-17 17:02:35,023] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: ge_tutorials_dag_with_ge.task_transform_data_in_db 2021-05-17T17:02:07.041650+00:00 [queued]>
[2021-05-17 17:02:35,037] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: ge_tutorials_dag_with_ge.task_transform_data_in_db 2021-05-17T17:02:07.041650+00:00 [queued]>
[2021-05-17 17:02:35,037] {{taskinstance.py:866}} INFO - 
--------------------------------------------------------------------------------
[2021-05-17 17:02:35,037] {{taskinstance.py:867}} INFO - Starting attempt 1 of 1
[2021-05-17 17:02:35,037] {{taskinstance.py:868}} INFO - 
--------------------------------------------------------------------------------
[2021-05-17 17:02:35,046] {{taskinstance.py:887}} INFO - Executing <Task(BashOperator): task_transform_data_in_db> on 2021-05-17T17:02:07.041650+00:00
[2021-05-17 17:02:35,049] {{standard_task_runner.py:53}} INFO - Started process 855 to run task
[2021-05-17 17:02:35,090] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: ge_tutorials_dag_with_ge.task_transform_data_in_db 2021-05-17T17:02:07.041650+00:00 [running]> e031efb82af6
[2021-05-17 17:02:35,106] {{bash_operator.py:82}} INFO - Tmp dir root location: 
 /tmp
[2021-05-17 17:02:35,107] {{bash_operator.py:105}} INFO - Temporary script location: /tmp/airflowtmp0lt4ylbu/task_transform_data_in_dboeetxzin
[2021-05-17 17:02:35,107] {{bash_operator.py:115}} INFO - Running command: dbt run --project-dir /usr/local/airflow/dbt
[2021-05-17 17:02:35,112] {{bash_operator.py:122}} INFO - Output:
**[2021-05-17 17:02:37,650] {{bash_operator.py:126}} INFO - Running with dbt=0.19.1
[2021-05-17 17:02:37,660] {{bash_operator.py:126}} INFO - Encountered an error while reading the project:**
[2021-05-17 17:02:37,661] {{bash_operator.py:126}} INFO -   ERROR: Runtime Error
[2021-05-17 17:02:37,661] {{bash_operator.py:126}} INFO -   Invalid config version: 1, expected 2
[2021-05-17 17:02:37,661] {{bash_operator.py:126}} INFO - 
[2021-05-17 17:02:37,661] {{bash_operator.py:126}} INFO - Error encountered in /usr/local/airflow/dbt/dbt_project.yml
[2021-05-17 17:02:37,665] {{bash_operator.py:126}} INFO - Encountered an error:
[2021-05-17 17:02:37,666] {{bash_operator.py:126}} INFO - Runtime Error
[2021-05-17 17:02:37,666] {{bash_operator.py:126}} INFO -   Could not run dbt
[2021-05-17 17:02:37,754] {{bash_operator.py:130}} INFO - Command exited with return code 2
[2021-05-17 17:02:37,761] {{taskinstance.py:1128}} ERROR - Bash command failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 966, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/bash_operator.py", line 134, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2021-05-17 17:02:37,763] {{taskinstance.py:1185}} INFO - Marking task as FAILED.dag_id=ge_tutorials_dag_with_ge, task_id=task_transform_data_in_db, execution_date=20210517T170207, start_date=20210517T170235, end_date=20210517T170237
[2021-05-17 17:02:45,018] {{logging_mixin.py:112}} INFO - [2021-05-17 17:02:45,017] {{local_task_job.py:103}} INFO - Task exited with return code 1
sleepo-chin commented 3 years ago

I find this error in dbt/dbt_project.yml we need to add this config-version: 2 because dbt version >0.19.0 default of config-version is 1.