search

greatexpectationslabs / ge_tutorials

Learn how to add data validation and documentation to a data pipeline built with dbt and Airflow.
165 stars 84 forks source link

Great Expectations Error in Airflow execution #8

Open JeroenSchmidt opened 4 years ago

JeroenSchmidt commented 4 years ago

The airflow (GE) DAG is unable to successfully execute because the configs appear to be incompatible with the version of GE that is being installed within the container. See error below;

*** Reading local file: /usr/local/airflow/logs/ge_tutorials_dag_with_ge/task_validate_source_data/2020-07-08T14:23:36.810518+00:00/1.log
[2020-07-08 14:23:40,135] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: ge_tutorials_dag_with_ge.task_validate_source_data 2020-07-08T14:23:36.810518+00:00 [queued]>
[2020-07-08 14:23:40,153] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: ge_tutorials_dag_with_ge.task_validate_source_data 2020-07-08T14:23:36.810518+00:00 [queued]>
[2020-07-08 14:23:40,153] {{taskinstance.py:866}} INFO - 
--------------------------------------------------------------------------------
[2020-07-08 14:23:40,153] {{taskinstance.py:867}} INFO - Starting attempt 1 of 1
[2020-07-08 14:23:40,153] {{taskinstance.py:868}} INFO - 
--------------------------------------------------------------------------------
[2020-07-08 14:23:40,164] {{taskinstance.py:887}} INFO - Executing <Task(PythonOperator): task_validate_source_data> on 2020-07-08T14:23:36.810518+00:00
[2020-07-08 14:23:40,168] {{standard_task_runner.py:53}} INFO - Started process 1006 to run task
[2020-07-08 14:23:40,211] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: ge_tutorials_dag_with_ge.task_validate_source_data 2020-07-08T14:23:36.810518+00:00 [running]> 26f657a26611
[2020-07-08 14:23:40,252] {{taskinstance.py:1128}} ERROR - You appear to have an invalid config version (1.0).
    The version number must be at least 2. Please see the migration guide at https://docs.greatexpectations.io/en/latest/how_to_guides/migrating_versions.html
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 966, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/ge_tutorials_dag_with_great_expectations.py", line 70, in validate_source_data
    context = ge.data_context.DataContext(great_expectations_context_path)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 2148, in __init__
    project_config = self._load_project_config()
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 2186, in _load_project_config
    return DataContextConfig.from_commented_map(config_dict)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/types/base.py", line 86, in from_commented_map
    config = dataContextConfigSchema.load(commented_map)
  File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 723, in load
    data, many=many, partial=partial, unknown=unknown, postprocess=True
  File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 886, in _do_load
    field_errors=field_errors,
  File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 1189, in _invoke_schema_validators
    partial=partial,
  File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 774, in _run_validator
    validator_func(output, partial=partial, many=many)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/types/base.py", line 298, in validate_schema
    data["config_version"], MINIMUM_SUPPORTED_CONFIG_VERSION
great_expectations.exceptions.UnsupportedConfigVersionError: You appear to have an invalid config version (1.0).
    The version number must be at least 2. Please see the migration guide at https://docs.greatexpectations.io/en/latest/how_to_guides/migrating_versions.html
[2020-07-08 14:23:40,254] {{taskinstance.py:1185}} INFO - Marking task as FAILED.dag_id=ge_tutorials_dag_with_ge, task_id=task_validate_source_data, execution_date=20200708T142336, start_date=20200708T142340, end_date=20200708T142340
[2020-07-08 14:23:50,113] {{logging_mixin.py:112}} INFO - [2020-07-08 14:23:50,113] {{local_task_job.py:103}} INFO - Task exited with return code 1
torsjonas commented 4 years ago

This looks like a duplicate of #7