[Bug] v0.20.0 version of dag-factory seems to be broken

[vishakseshadri]

DAG Factory version

0.20.0

airflow version

2.8.4

Python version

3.11

Deployment

Astronomer

Deployment details

This is on hosted astro cloud version 10.6.0

What happened?

The following error popped up with a new deployment and we noticed dag-factory to have updated -

` Broken DAG: [/usr/local/airflow/dags/dag_factory.py] Traceback (most recent call last): File "/usr/local/lib/python3.11/socket.py", line 827, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) httpcore.ConnectError: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ConnectError: [Errno -3] Temporary failure in name resolution `

Pinning the package to 0.19.0 seems to fix the issue

Relevant log output

No response

How to reproduce

Do a deploy without a pinned version of dag-factory

Anything else :)?

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Contact Details

No response

[vishakseshadri]

This may be an issue specific to version 10.x

[tatiana]

HI @vishakseshadri thank you very much for reporting this issue. We'll investigate this issue. Could you share a reproducible DAG?

[rorydonaldson]

Hello, was seeing the same issue with 0.20.0, and reverting to 0.19.0 fixed it. Python 3.10, airflow 2.4.3.

Example dag generator:

import os
from airflow import DAG 
import dagfactory

config_files = ["ctm/ctm_daily_config.yml"]

for config_file in config_files:
    full_config_file = os.path.join(
        os.environ["AIRFLOW_HOME"], "dags", "reports", config_file
    )
    dag_factory = dagfactory.DagFactory(full_config_file)
    dag_factory.clean_dags(globals())
    dag_factory.generate_dags(globals())

[pankajkoti]

hi @rorydonaldson @vishakseshadri

The DAG shared or the error log do not provide enough details to debug this much. Can you please share some more details from your ctm/ctm_daily_config.yml -- especially what operators it is trying to use, what all params are getting passed.

Just taking a wild guess if it is using the HTTPSensor sensor could you then please update the operator path in your YAML similar to the changes shown in PR https://github.com/astronomer/dag-factory/pull/265/files

[vishakseshadri]

Here's an example of our config file

default:
  default_args:
    owner: ddeely
    start_date: 2024-10-25
    retries: 0
    retry_delay_sec: 300
    email: []
    email_on_failure: False
    email_on_retry: False
    timezone: America/Chicago
    op_kwargs: {"config_path": "/usr/local/airflow/include/dag-config/path/to/config/dag.yml"}

  concurrency: 1
  max_active_runs: 1
  dagrun_timeout_sec: 4800
  catchup: False
  default_view: tree
  orientation: LR
  schedule_interval: None
  render_template_as_native_obj: False

check_name:
  schedule_interval: 0 8 * * *
  description: some description
  tasks:
    load_configs_and_run_query:
      operator: plugins.query_operator.QueryOperator
      job_configs:
        postgres_conn_id: us_replica
        sql: '/usr/local/airflow/include/dag-config/path/to/query/query.sql'
        email_attach_results: True
        jira_attach_results: True
    check_results:
      operator: airflow.operators.python.BranchPythonOperator
      python_callable_name: branch_func
      python_callable_file: /usr/local/airflow/plugins/branch.py
      dependencies: [load_configs_and_run_query]
    escalations_required_no:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: 'echo No Results Found. No Escalation Required.'
      dependencies: [check_results]
    email_escalation:
      email_args:
        senderEmail: astronomer@enova.com
        subject: Unwanted Results for {{ dag.dag_id }} - Environment - {{ task_instance.xcom_pull(task_ids='load_configs_and_run_query', key='current_env') }}
        body: "{{ task_instance.xcom_pull(task_ids='load_configs_and_run_query', key='count_results') }} results found for {{ dag_run.dag_id }}"
        recipientEmail:
          - "{{'recipients@something.com' if task_instance.xcom_pull(task_ids='load_configs_and_run_query', key='current_env') == 'production' else 'recipients@something.com'}}"
        files: []
      operator: plugins.mail_operator.MailOperator
      dependencies: [check_results]
    jira_escalation:
      jira_method_args: {
        'project': { 'id': '21660'},
        'priority': {'id': '3'}, # 1-Blocker, 2-Critical, 3-Major, 4-Minor, 5-Trivial
        'issuetype': {'id': '1'}, # 1-general
        'description':  "@username"
        }
      operator: plugins.jira_operator.JiraOperator
      dependencies: [check_results]

And this is our dag_factory.py

from dagfactory import load_yaml_dags
from airflow.timetables.trigger import CronTriggerTimetable

load_yaml_dags(globals_dict=globals(), dags_folder="/usr/local/airflow/include/dag-config", suffix=['dags.yml', 'dags.yaml', 'dag.yml', 'dag.yaml'])

[pankajkoti]

Thanks a lot @vishakseshadri for sharing this. I will try to reproduce this & come back with our findings on this one soon.