astronomer / airflow-provider-great-expectations

Great Expectations Airflow operator
http://greatexpectations.io
Apache License 2.0
159 stars 54 forks source link

Operator won't work in 0.2.1 #62

Closed patricker closed 1 year ago

patricker commented 1 year ago

With the new 0.2.1 release of the Great Expectations Operator we've been getting the below error. We reverted to 0.2.0 and the error went away.

Error:

[2022-11-17, 01:58:38 UTC] {{abstractoperator.py:175}} ERROR - Failed to resolve template field 'data_context_root_dir'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 173, in resolve_template_files
    setattr(self, field, env.loader.get_source(env, content)[0])  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/jinja2/loaders.py", line 218, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: /usr/src/app/great_expectations

Code:

    ge_data_unit_test_op = GreatExpectationsOperator(
        task_id="test_package",
        data_context_root_dir=ge_root_dir,
        checkpoint_name=test_package,
        checkpoint_kwargs={
            "batch_request": {
                "datasource_name": TARGET_PROJECT_ID,
                "data_asset_name": data_asset_name,
            }
        },
    )
kaxil commented 1 year ago

Try with the following, we will fix it in 0.2.2

class CustomGreatExpectationsOperator:
      template_fields = (
        "run_name",
        "conn_id",
        "checkpoint_name",
        "checkpoint_kwargs",
        "query_to_validate"
    )

    ge_data_unit_test_op = CustomGreatExpectationsOperator(
        task_id="test_package",
        data_context_root_dir=ge_root_dir,
        checkpoint_name=test_package,
        checkpoint_kwargs={
            "batch_request": {
                "datasource_name": TARGET_PROJECT_ID,
                "data_asset_name": data_asset_name,
            }
        },
    )
kaxil commented 1 year ago

This was fixed in https://github.com/astronomer/airflow-provider-great-expectations/commit/3d07ab9d2708d5e32ffff6da7cc03a1b8c10d8a8 and released in 0.2.2

Can you please upgrade and try it out?