search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.12k stars 14.31k forks source link

Tasks taking long to get executed #36795

Closed SalvaChiLlo closed 10 months ago

SalvaChiLlo commented 10 months ago

Apache Airflow version

2.8.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Tasks are taking to long to get executed, workers receive the order to execute a task but once the workers execute the command, it takes several seconds for the actual task to get executed. For example, example_complex dag is taking between a minute and a half and two minutes to complete, but the actual task execution time for each of the tasks is less than a second.

I have custom workers that receive tasks from kafka and execute them.

I'll append a few logs:

What you think should happen instead?

Tasks take less than a second to get actually executed, but the total amount of time executing the dag is much bigger than expected.

How to reproduce

I just try to execute the example_complex dag although the described behaviour is the same for every other dag.

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

pip freeze | grep providers apache-airflow-providers-amazon==8.16.0 apache-airflow-providers-celery==3.5.0 apache-airflow-providers-cncf-kubernetes==7.11.0 apache-airflow-providers-common-io==1.1.0 apache-airflow-providers-common-sql==1.9.0 apache-airflow-providers-docker==3.8.2 apache-airflow-providers-elasticsearch==5.3.0 apache-airflow-providers-ftp==3.7.0 apache-airflow-providers-google==10.12.0 apache-airflow-providers-grpc==3.4.0 apache-airflow-providers-hashicorp==3.6.0 apache-airflow-providers-http==4.8.0 apache-airflow-providers-imap==3.5.0 apache-airflow-providers-microsoft-azure==8.4.0 apache-airflow-providers-mysql==5.5.0 apache-airflow-providers-odbc==4.2.0 apache-airflow-providers-openlineage==1.3.0 apache-airflow-providers-postgres==5.9.0 apache-airflow-providers-redis==3.5.0 apache-airflow-providers-sendgrid==3.4.0 apache-airflow-providers-sftp==4.8.0 apache-airflow-providers-slack==8.5.0 apache-airflow-providers-snowflake==5.2.0 apache-airflow-providers-sqlite==3.6.0 apache-airflow-providers-ssh==3.9.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

My deployment is pretty similar to the one using docker compose but a big difference is that I have a Kafka executor and workers, for discarding I made several tests to ensure that they are not introducing unexpected delays.

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 10 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

SalvaChiLlo commented 10 months ago 
Command executed successfully: airflow tasks run example_complex delete_entry_group manual__2024-01-15T15:40:09.891366+00:00 --local --subdir /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_complex.py
[2024-01-15T15:41:29.620+0000] {settings.py:234} DEBUG - Setting up DB connection pool (PID 18025)
[2024-01-15T15:41:29.621+0000] {settings.py:335} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=0, max_overflow=-1, pool_recycle=1800, pid=18025
[2024-01-15T15:41:29.656+0000] {cli_action_loggers.py:41} DEBUG - Adding <function default_action_log at 0x7f0d4b95da80> to pre execution callback
[2024-01-15T15:41:29.660+0000] {executor_loader.py:143} DEBUG - The executor name looks like the plugin path (executor_name=kafka_executor_plugin.KafkaExecutor). Trying to import a executor from a plugin
[2024-01-15T15:41:29.668+0000] {plugins_manager.py:328} DEBUG - Loading plugins
[2024-01-15T15:41:29.668+0000] {plugins_manager.py:254} DEBUG - Loading plugins from directory: /opt/airflow/plugins
[2024-01-15T15:41:29.921+0000] {serde.py:339} DEBUG - registering decimal.Decimal for serialization
[2024-01-15T15:41:29.922+0000] {serde.py:346} DEBUG - registering decimal.Decimal for deserialization
[2024-01-15T15:41:29.922+0000] {serde.py:339} DEBUG - registering builtins.frozenset for serialization
[2024-01-15T15:41:29.922+0000] {serde.py:339} DEBUG - registering builtins.set for serialization
[2024-01-15T15:41:29.922+0000] {serde.py:339} DEBUG - registering builtins.tuple for serialization
[2024-01-15T15:41:29.922+0000] {serde.py:346} DEBUG - registering builtins.frozenset for deserialization
[2024-01-15T15:41:29.922+0000] {serde.py:346} DEBUG - registering builtins.set for deserialization
[2024-01-15T15:41:29.923+0000] {serde.py:346} DEBUG - registering builtins.tuple for deserialization
[2024-01-15T15:41:29.923+0000] {serde.py:354} DEBUG - registering builtins.frozenset for stringifying
[2024-01-15T15:41:29.923+0000] {serde.py:354} DEBUG - registering builtins.set for stringifying
[2024-01-15T15:41:29.923+0000] {serde.py:354} DEBUG - registering builtins.tuple for stringifying
[2024-01-15T15:41:29.923+0000] {serde.py:339} DEBUG - registering datetime.date for serialization
[2024-01-15T15:41:29.924+0000] {serde.py:339} DEBUG - registering datetime.datetime for serialization
[2024-01-15T15:41:29.924+0000] {serde.py:339} DEBUG - registering datetime.timedelta for serialization
[2024-01-15T15:41:29.924+0000] {serde.py:339} DEBUG - registering pendulum.datetime.DateTime for serialization
[2024-01-15T15:41:29.924+0000] {serde.py:346} DEBUG - registering datetime.date for deserialization
[2024-01-15T15:41:29.924+0000] {serde.py:346} DEBUG - registering datetime.datetime for deserialization
[2024-01-15T15:41:29.924+0000] {serde.py:346} DEBUG - registering datetime.timedelta for deserialization
[2024-01-15T15:41:29.924+0000] {serde.py:346} DEBUG - registering pendulum.datetime.DateTime for deserialization
[2024-01-15T15:41:29.924+0000] {serde.py:339} DEBUG - registering deltalake.table.DeltaTable for serialization
[2024-01-15T15:41:29.925+0000] {serde.py:346} DEBUG - registering deltalake.table.DeltaTable for deserialization
[2024-01-15T15:41:29.925+0000] {serde.py:354} DEBUG - registering deltalake.table.DeltaTable for stringifying
[2024-01-15T15:41:29.925+0000] {serde.py:339} DEBUG - registering pyiceberg.table.Table for serialization
[2024-01-15T15:41:29.925+0000] {serde.py:346} DEBUG - registering pyiceberg.table.Table for deserialization
[2024-01-15T15:41:29.925+0000] {serde.py:354} DEBUG - registering pyiceberg.table.Table for stringifying
[2024-01-15T15:41:29.925+0000] {serde.py:339} DEBUG - registering kubernetes.client.models.v1_resource_requirements.V1ResourceRequirements for serialization
[2024-01-15T15:41:29.926+0000] {serde.py:339} DEBUG - registering kubernetes.client.models.v1_pod.V1Pod for serialization
[2024-01-15T15:41:29.926+0000] {serde.py:339} DEBUG - registering numpy.int8 for serialization
[2024-01-15T15:41:29.926+0000] {serde.py:339} DEBUG - registering numpy.int16 for serialization
[2024-01-15T15:41:29.926+0000] {serde.py:339} DEBUG - registering numpy.int32 for serialization
[2024-01-15T15:41:29.926+0000] {serde.py:339} DEBUG - registering numpy.int64 for serialization
[2024-01-15T15:41:29.926+0000] {serde.py:339} DEBUG - registering numpy.uint8 for serialization
[2024-01-15T15:41:29.926+0000] {serde.py:339} DEBUG - registering numpy.uint16 for serialization
[2024-01-15T15:41:29.926+0000] {serde.py:339} DEBUG - registering numpy.uint32 for serialization
[2024-01-15T15:41:29.927+0000] {serde.py:339} DEBUG - registering numpy.uint64 for serialization
[2024-01-15T15:41:29.927+0000] {serde.py:339} DEBUG - registering numpy.bool_ for serialization
[2024-01-15T15:41:29.927+0000] {serde.py:339} DEBUG - registering numpy.float64 for serialization
[2024-01-15T15:41:29.927+0000] {serde.py:339} DEBUG - registering numpy.float16 for serialization
[2024-01-15T15:41:29.927+0000] {serde.py:339} DEBUG - registering numpy.complex128 for serialization
[2024-01-15T15:41:29.927+0000] {serde.py:339} DEBUG - registering numpy.complex64 for serialization
[2024-01-15T15:41:29.927+0000] {serde.py:346} DEBUG - registering numpy.int8 for deserialization
[2024-01-15T15:41:29.927+0000] {serde.py:346} DEBUG - registering numpy.int16 for deserialization
[2024-01-15T15:41:29.927+0000] {serde.py:346} DEBUG - registering numpy.int32 for deserialization
[2024-01-15T15:41:29.927+0000] {serde.py:346} DEBUG - registering numpy.int64 for deserialization
[2024-01-15T15:41:29.927+0000] {serde.py:346} DEBUG - registering numpy.uint8 for deserialization
[2024-01-15T15:41:29.927+0000] {serde.py:346} DEBUG - registering numpy.uint16 for deserialization
[2024-01-15T15:41:29.927+0000] {serde.py:346} DEBUG - registering numpy.uint32 for deserialization
[2024-01-15T15:41:29.928+0000] {serde.py:346} DEBUG - registering numpy.uint64 for deserialization
[2024-01-15T15:41:29.928+0000] {serde.py:346} DEBUG - registering numpy.bool_ for deserialization
[2024-01-15T15:41:29.928+0000] {serde.py:346} DEBUG - registering numpy.float64 for deserialization
[2024-01-15T15:41:29.928+0000] {serde.py:346} DEBUG - registering numpy.float16 for deserialization
[2024-01-15T15:41:29.928+0000] {serde.py:346} DEBUG - registering numpy.complex128 for deserialization
[2024-01-15T15:41:29.928+0000] {serde.py:346} DEBUG - registering numpy.complex64 for deserialization
[2024-01-15T15:41:29.928+0000] {serde.py:339} DEBUG - registering pandas.core.frame.DataFrame for serialization
[2024-01-15T15:41:29.928+0000] {serde.py:346} DEBUG - registering pandas.core.frame.DataFrame for deserialization
[2024-01-15T15:41:29.929+0000] {serde.py:339} DEBUG - registering pendulum.tz.timezone.FixedTimezone for serialization
[2024-01-15T15:41:29.929+0000] {serde.py:339} DEBUG - registering pendulum.tz.timezone.Timezone for serialization
[2024-01-15T15:41:29.929+0000] {serde.py:339} DEBUG - registering zoneinfo.ZoneInfo for serialization
[2024-01-15T15:41:29.929+0000] {serde.py:346} DEBUG - registering pendulum.tz.timezone.FixedTimezone for deserialization
[2024-01-15T15:41:29.929+0000] {serde.py:346} DEBUG - registering pendulum.tz.timezone.Timezone for deserialization
[2024-01-15T15:41:29.929+0000] {serde.py:346} DEBUG - registering zoneinfo.ZoneInfo for deserialization
[2024-01-15T15:41:29.929+0000] {serde.py:357} DEBUG - loading serializers took 0.009 seconds
[2024-01-15T15:41:29.991+0000] {plugins_manager.py:268} DEBUG - Importing plugin module /opt/airflow/plugins/kafka_executor_plugin.py
[2024-01-15T15:41:29.991+0000] {plugins_manager.py:234} DEBUG - Loading plugins from entrypoints
[2024-01-15T15:41:30.016+0000] {plugins_manager.py:237} DEBUG - Importing entry_point plugin openlineage
[2024-01-15T15:41:30.492+0000] {plugins_manager.py:346} DEBUG - Loading 1 plugin(s) took 0.82 seconds
[2024-01-15T15:41:30.492+0000] {plugins_manager.py:477} DEBUG - Integrate executor plugins
[2024-01-15T15:41:31.182+0000] {cli_action_loggers.py:67} DEBUG - Calling callbacks: [<function default_action_log at 0x7f0d4b95da80>]
[2024-01-15T15:41:31.222+0000] {plugins_manager.py:322} DEBUG - Plugins are already loaded. Skipping.
[2024-01-15T15:41:31.223+0000] {dagbag.py:538} INFO - Filling up the DagBag from /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_complex.py
[2024-01-15T15:41:31.242+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_complex.py
[2024-01-15T15:41:31.257+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_complex>
[2024-01-15T15:41:31.257+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_labels.py
[2024-01-15T15:41:31.259+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.261+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.262+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_labels>
[2024-01-15T15:41:31.262+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_python_operator.py
[2024-01-15T15:41:31.277+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_python_operator>
[2024-01-15T15:41:31.277+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_sla_dag.py
[2024-01-15T15:41:31.287+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.289+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_sla_dag>
[2024-01-15T15:41:31.289+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_subdag_operator.py
[2024-01-15T15:41:31.291+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.392+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.397+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_subdag_operator.section-1>
[2024-01-15T15:41:31.397+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_subdag_operator.section-2>
[2024-01-15T15:41:31.397+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_subdag_operator>
[2024-01-15T15:41:31.398+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
[2024-01-15T15:41:31.400+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_dynamic_task_mapping_with_no_taskflow_operators>
[2024-01-15T15:41:31.401+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_xcom.py
[2024-01-15T15:41:31.403+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_xcom>
[2024-01-15T15:41:31.404+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_kubernetes_executor.py
[2024-01-15T15:41:31.411+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_kubernetes_executor>
[2024-01-15T15:41:31.412+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_task_group_decorator.py
[2024-01-15T15:41:31.420+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_task_group_decorator>
[2024-01-15T15:41:31.421+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_operator_decorator.py
[2024-01-15T15:41:31.422+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.432+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.433+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_python_operator_decorator>
[2024-01-15T15:41:31.434+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_datasets.py
[2024-01-15T15:41:31.435+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.436+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_consumes_1>
[2024-01-15T15:41:31.437+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_consumes_unknown_never_scheduled>
[2024-01-15T15:41:31.437+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.438+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_produces_1>
[2024-01-15T15:41:31.438+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_consumes_1_and_2>
[2024-01-15T15:41:31.438+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_consumes_1_never_scheduled>
[2024-01-15T15:41:31.438+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_produces_2>
[2024-01-15T15:41:31.438+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_params_ui_tutorial.py
[2024-01-15T15:41:31.442+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_params_ui_tutorial>
[2024-01-15T15:41:31.443+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_sensors.py
[2024-01-15T15:41:31.452+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_sensors>
[2024-01-15T15:41:31.453+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_short_circuit_operator.py
[2024-01-15T15:41:31.455+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_short_circuit_operator>
[2024-01-15T15:41:31.456+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_operator.py
[2024-01-15T15:41:31.457+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.630+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.631+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_operator>
[2024-01-15T15:41:31.632+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_external_task_marker_dag.py
[2024-01-15T15:41:31.636+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_external_task_marker_child>
[2024-01-15T15:41:31.636+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_external_task_marker_parent>
[2024-01-15T15:41:31.636+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_local_kubernetes_executor.py
[2024-01-15T15:41:31.638+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_local_kubernetes_executor>
[2024-01-15T15:41:31.639+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_trigger_target_dag.py
[2024-01-15T15:41:31.640+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_trigger_target_dag>
[2024-01-15T15:41:31.640+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_latest_only_with_trigger.py
[2024-01-15T15:41:31.643+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: latest_only_with_trigger>
[2024-01-15T15:41:31.643+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_short_circuit_decorator.py
[2024-01-15T15:41:31.647+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_short_circuit_decorator>
[2024-01-15T15:41:31.647+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial.py
[2024-01-15T15:41:31.648+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial>
[2024-01-15T15:41:31.649+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial_dag.py
[2024-01-15T15:41:31.650+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial_dag>
[2024-01-15T15:41:31.650+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_time_delta_sensor_async.py
[2024-01-15T15:41:31.651+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_time_delta_sensor_async>
[2024-01-15T15:41:31.652+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_xcomargs.py
[2024-01-15T15:41:31.655+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_xcom_args_with_operators>
[2024-01-15T15:41:31.655+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_xcom_args>
[2024-01-15T15:41:31.655+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_python_dop_operator_3.py
[2024-01-15T15:41:31.656+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.658+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_dop_operator_v3>
[2024-01-15T15:41:31.659+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_latest_only.py
[2024-01-15T15:41:31.660+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: latest_only>
[2024-01-15T15:41:31.660+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_day_of_week_operator.py
[2024-01-15T15:41:31.661+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.663+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.663+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_weekday_branch_operator>
[2024-01-15T15:41:31.664+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_python_decorator.py
[2024-01-15T15:41:31.668+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_python_decorator>
[2024-01-15T15:41:31.669+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_dynamic_task_mapping.py
[2024-01-15T15:41:31.670+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_dynamic_task_mapping>
[2024-01-15T15:41:31.670+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_skip_dag.py
[2024-01-15T15:41:31.672+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_skip_dag>
[2024-01-15T15:41:31.673+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial_objectstorage.py
[2024-01-15T15:41:31.740+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial_objectstorage>
[2024-01-15T15:41:31.741+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_setup_teardown.py
[2024-01-15T15:41:31.743+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_setup_teardown>
[2024-01-15T15:41:31.743+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_datetime_operator.py
[2024-01-15T15:41:31.744+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.745+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.746+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.747+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.748+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_datetime_operator_2>
[2024-01-15T15:41:31.748+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.749+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_datetime_operator>
[2024-01-15T15:41:31.749+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.750+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_datetime_operator_3>
[2024-01-15T15:41:31.750+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_nested_branch_dag.py
[2024-01-15T15:41:31.751+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.753+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.754+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_nested_branch_dag>
[2024-01-15T15:41:31.754+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_trigger_controller_dag.py
[2024-01-15T15:41:31.756+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_trigger_controller_dag>
[2024-01-15T15:41:31.756+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_bash_operator.py
[2024-01-15T15:41:31.757+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.760+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_bash_operator>
[2024-01-15T15:41:31.761+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_bash_operator>
[2024-01-15T15:41:31.761+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_sensor_decorator.py
[2024-01-15T15:41:31.763+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_sensor_decorator>
[2024-01-15T15:41:31.763+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_dag_decorator.py
[2024-01-15T15:41:31.841+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_dag_decorator>
[2024-01-15T15:41:31.842+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial_taskflow_api_virtualenv.py
[2024-01-15T15:41:31.844+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial_taskflow_api_virtualenv>
[2024-01-15T15:41:31.844+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_passing_params_via_test_command.py
[2024-01-15T15:41:31.845+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:31.848+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_passing_params_via_test_command>
[2024-01-15T15:41:31.848+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_task_group.py
[2024-01-15T15:41:31.850+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_task_group>
[2024-01-15T15:41:31.850+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_params_trigger_ui.py
[2024-01-15T15:41:31.855+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_params_trigger_ui>
[2024-01-15T15:41:31.855+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_setup_teardown_taskflow.py
[2024-01-15T15:41:31.859+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_setup_teardown_taskflow>
[2024-01-15T15:41:31.859+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial_taskflow_api.py
[2024-01-15T15:41:31.861+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial_taskflow_api>
[2024-01-15T15:41:31.861+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/plugins/event_listener.py
[2024-01-15T15:41:31.862+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/plugins/workday.py
[2024-01-15T15:41:32.155+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/plugins/listener_plugin.py
[2024-01-15T15:41:32.157+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/subdags/subdag.py
[2024-01-15T15:41:32.199+0000] {plugins_manager.py:322} DEBUG - Plugins are already loaded. Skipping.
[2024-01-15T15:41:32.200+0000] {plugins_manager.py:506} DEBUG - Integrate DAG plugins
[2024-01-15T15:41:32.204+0000] {taskinstance.py:980} DEBUG - previous_execution_date was called
Changing /home/airflow/sources/logs/dag_id=example_complex/run_id=manual__2024-01-15T15:40:09.891366+00:00/task_id=delete_entry_group permission to 509
[2024-01-15T15:41:32.208+0000] {task_command.py:423} INFO - Running <TaskInstance: example_complex.delete_entry_group manual__2024-01-15T15:40:09.891366+00:00 [queued]> on host kd-073023-bdf8b54b-ad46159c-deployment-dd6d9d4d6-85nxx
[2024-01-15T15:41:32.208+0000] {settings.py:371} DEBUG - Disposing DB connection pool (PID 18025)
[2024-01-15T15:41:32.209+0000] {settings.py:234} DEBUG - Setting up DB connection pool (PID 18025)
[2024-01-15T15:41:32.209+0000] {settings.py:303} DEBUG - settings.prepare_engine_args(): Using NullPool
[2024-01-15T15:41:33.749+0000] {cli_action_loggers.py:85} DEBUG - Calling callbacks: []
[2024-01-15T15:41:33.750+0000] {settings.py:371} DEBUG - Disposing DB connection pool (PID 18025)

Task example_complex,delete_entry_group,manual__2024-01-15T15:40:09.891366+00:00,1,-1 completed successfully
Command executed successfully: airflow tasks run example_complex delete_entry manual__2024-01-15T15:40:09.891366+00:00 --local --subdir /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_complex.py
[2024-01-15T15:41:34.810+0000] {settings.py:234} DEBUG - Setting up DB connection pool (PID 18061)
[2024-01-15T15:41:34.811+0000] {settings.py:335} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=0, max_overflow=-1, pool_recycle=1800, pid=18061
[2024-01-15T15:41:34.844+0000] {cli_action_loggers.py:41} DEBUG - Adding <function default_action_log at 0x7f66b902da80> to pre execution callback
[2024-01-15T15:41:34.848+0000] {executor_loader.py:143} DEBUG - The executor name looks like the plugin path (executor_name=kafka_executor_plugin.KafkaExecutor). Trying to import a executor from a plugin
[2024-01-15T15:41:34.855+0000] {plugins_manager.py:328} DEBUG - Loading plugins
[2024-01-15T15:41:34.855+0000] {plugins_manager.py:254} DEBUG - Loading plugins from directory: /opt/airflow/plugins
[2024-01-15T15:41:35.076+0000] {serde.py:339} DEBUG - registering decimal.Decimal for serialization
[2024-01-15T15:41:35.076+0000] {serde.py:346} DEBUG - registering decimal.Decimal for deserialization
[2024-01-15T15:41:35.077+0000] {serde.py:339} DEBUG - registering builtins.frozenset for serialization
[2024-01-15T15:41:35.077+0000] {serde.py:339} DEBUG - registering builtins.set for serialization
[2024-01-15T15:41:35.077+0000] {serde.py:339} DEBUG - registering builtins.tuple for serialization
[2024-01-15T15:41:35.077+0000] {serde.py:346} DEBUG - registering builtins.frozenset for deserialization
[2024-01-15T15:41:35.077+0000] {serde.py:346} DEBUG - registering builtins.set for deserialization
[2024-01-15T15:41:35.077+0000] {serde.py:346} DEBUG - registering builtins.tuple for deserialization
[2024-01-15T15:41:35.077+0000] {serde.py:354} DEBUG - registering builtins.frozenset for stringifying
[2024-01-15T15:41:35.077+0000] {serde.py:354} DEBUG - registering builtins.set for stringifying
[2024-01-15T15:41:35.078+0000] {serde.py:354} DEBUG - registering builtins.tuple for stringifying
[2024-01-15T15:41:35.078+0000] {serde.py:339} DEBUG - registering datetime.date for serialization
[2024-01-15T15:41:35.078+0000] {serde.py:339} DEBUG - registering datetime.datetime for serialization
[2024-01-15T15:41:35.078+0000] {serde.py:339} DEBUG - registering datetime.timedelta for serialization
[2024-01-15T15:41:35.078+0000] {serde.py:339} DEBUG - registering pendulum.datetime.DateTime for serialization
[2024-01-15T15:41:35.078+0000] {serde.py:346} DEBUG - registering datetime.date for deserialization
[2024-01-15T15:41:35.079+0000] {serde.py:346} DEBUG - registering datetime.datetime for deserialization
[2024-01-15T15:41:35.079+0000] {serde.py:346} DEBUG - registering datetime.timedelta for deserialization
[2024-01-15T15:41:35.079+0000] {serde.py:346} DEBUG - registering pendulum.datetime.DateTime for deserialization
[2024-01-15T15:41:35.079+0000] {serde.py:339} DEBUG - registering deltalake.table.DeltaTable for serialization
[2024-01-15T15:41:35.079+0000] {serde.py:346} DEBUG - registering deltalake.table.DeltaTable for deserialization
[2024-01-15T15:41:35.079+0000] {serde.py:354} DEBUG - registering deltalake.table.DeltaTable for stringifying
[2024-01-15T15:41:35.079+0000] {serde.py:339} DEBUG - registering pyiceberg.table.Table for serialization
[2024-01-15T15:41:35.080+0000] {serde.py:346} DEBUG - registering pyiceberg.table.Table for deserialization
[2024-01-15T15:41:35.080+0000] {serde.py:354} DEBUG - registering pyiceberg.table.Table for stringifying
[2024-01-15T15:41:35.080+0000] {serde.py:339} DEBUG - registering kubernetes.client.models.v1_resource_requirements.V1ResourceRequirements for serialization
[2024-01-15T15:41:35.080+0000] {serde.py:339} DEBUG - registering kubernetes.client.models.v1_pod.V1Pod for serialization
[2024-01-15T15:41:35.080+0000] {serde.py:339} DEBUG - registering numpy.int8 for serialization
[2024-01-15T15:41:35.080+0000] {serde.py:339} DEBUG - registering numpy.int16 for serialization
[2024-01-15T15:41:35.080+0000] {serde.py:339} DEBUG - registering numpy.int32 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.int64 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.uint8 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.uint16 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.uint32 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.uint64 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.bool_ for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.float64 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.float16 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.complex128 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:339} DEBUG - registering numpy.complex64 for serialization
[2024-01-15T15:41:35.081+0000] {serde.py:346} DEBUG - registering numpy.int8 for deserialization
[2024-01-15T15:41:35.081+0000] {serde.py:346} DEBUG - registering numpy.int16 for deserialization
[2024-01-15T15:41:35.081+0000] {serde.py:346} DEBUG - registering numpy.int32 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.int64 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.uint8 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.uint16 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.uint32 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.uint64 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.bool_ for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.float64 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.float16 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.complex128 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:346} DEBUG - registering numpy.complex64 for deserialization
[2024-01-15T15:41:35.082+0000] {serde.py:339} DEBUG - registering pandas.core.frame.DataFrame for serialization
[2024-01-15T15:41:35.083+0000] {serde.py:346} DEBUG - registering pandas.core.frame.DataFrame for deserialization
[2024-01-15T15:41:35.083+0000] {serde.py:339} DEBUG - registering pendulum.tz.timezone.FixedTimezone for serialization
[2024-01-15T15:41:35.083+0000] {serde.py:339} DEBUG - registering pendulum.tz.timezone.Timezone for serialization
[2024-01-15T15:41:35.083+0000] {serde.py:339} DEBUG - registering zoneinfo.ZoneInfo for serialization
[2024-01-15T15:41:35.083+0000] {serde.py:346} DEBUG - registering pendulum.tz.timezone.FixedTimezone for deserialization
[2024-01-15T15:41:35.083+0000] {serde.py:346} DEBUG - registering pendulum.tz.timezone.Timezone for deserialization
[2024-01-15T15:41:35.083+0000] {serde.py:346} DEBUG - registering zoneinfo.ZoneInfo for deserialization
[2024-01-15T15:41:35.083+0000] {serde.py:357} DEBUG - loading serializers took 0.008 seconds
[2024-01-15T15:41:35.142+0000] {plugins_manager.py:268} DEBUG - Importing plugin module /opt/airflow/plugins/kafka_executor_plugin.py
[2024-01-15T15:41:35.142+0000] {plugins_manager.py:234} DEBUG - Loading plugins from entrypoints
[2024-01-15T15:41:35.165+0000] {plugins_manager.py:237} DEBUG - Importing entry_point plugin openlineage
[2024-01-15T15:41:35.567+0000] {plugins_manager.py:346} DEBUG - Loading 1 plugin(s) took 0.71 seconds
[2024-01-15T15:41:35.567+0000] {plugins_manager.py:477} DEBUG - Integrate executor plugins
[2024-01-15T15:41:36.278+0000] {cli_action_loggers.py:67} DEBUG - Calling callbacks: [<function default_action_log at 0x7f66b902da80>]
[2024-01-15T15:41:36.315+0000] {plugins_manager.py:322} DEBUG - Plugins are already loaded. Skipping.
[2024-01-15T15:41:36.315+0000] {dagbag.py:538} INFO - Filling up the DagBag from /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_complex.py
[2024-01-15T15:41:36.330+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_complex.py
[2024-01-15T15:41:36.346+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_complex>
[2024-01-15T15:41:36.347+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_labels.py
[2024-01-15T15:41:36.348+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.351+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.351+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_labels>
[2024-01-15T15:41:36.352+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_python_operator.py
[2024-01-15T15:41:36.368+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_python_operator>
[2024-01-15T15:41:36.369+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_sla_dag.py
[2024-01-15T15:41:36.380+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.382+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_sla_dag>
[2024-01-15T15:41:36.382+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_subdag_operator.py
[2024-01-15T15:41:36.384+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.489+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.493+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_subdag_operator.section-1>
[2024-01-15T15:41:36.493+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_subdag_operator.section-2>
[2024-01-15T15:41:36.493+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_subdag_operator>
[2024-01-15T15:41:36.494+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_dynamic_task_mapping_with_no_taskflow_operators.py
[2024-01-15T15:41:36.496+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_dynamic_task_mapping_with_no_taskflow_operators>
[2024-01-15T15:41:36.496+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_xcom.py
[2024-01-15T15:41:36.499+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_xcom>
[2024-01-15T15:41:36.499+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_kubernetes_executor.py
[2024-01-15T15:41:36.505+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_kubernetes_executor>
[2024-01-15T15:41:36.506+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_task_group_decorator.py
[2024-01-15T15:41:36.513+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_task_group_decorator>
[2024-01-15T15:41:36.514+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_operator_decorator.py
[2024-01-15T15:41:36.515+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.525+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.525+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_python_operator_decorator>
[2024-01-15T15:41:36.526+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_datasets.py
[2024-01-15T15:41:36.526+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.528+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_produces_2>
[2024-01-15T15:41:36.529+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.529+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_produces_1>
[2024-01-15T15:41:36.529+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_consumes_unknown_never_scheduled>
[2024-01-15T15:41:36.529+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_consumes_1_never_scheduled>
[2024-01-15T15:41:36.530+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_consumes_1>
[2024-01-15T15:41:36.530+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: dataset_consumes_1_and_2>
[2024-01-15T15:41:36.530+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_params_ui_tutorial.py
[2024-01-15T15:41:36.533+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_params_ui_tutorial>
[2024-01-15T15:41:36.534+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_sensors.py
[2024-01-15T15:41:36.542+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_sensors>
[2024-01-15T15:41:36.542+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_short_circuit_operator.py
[2024-01-15T15:41:36.544+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_short_circuit_operator>
[2024-01-15T15:41:36.545+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_operator.py
[2024-01-15T15:41:36.546+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.660+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.661+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_operator>
[2024-01-15T15:41:36.662+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_external_task_marker_dag.py
[2024-01-15T15:41:36.665+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_external_task_marker_parent>
[2024-01-15T15:41:36.665+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_external_task_marker_child>
[2024-01-15T15:41:36.666+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_local_kubernetes_executor.py
[2024-01-15T15:41:36.667+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_local_kubernetes_executor>
[2024-01-15T15:41:36.668+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_trigger_target_dag.py
[2024-01-15T15:41:36.669+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_trigger_target_dag>
[2024-01-15T15:41:36.669+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_latest_only_with_trigger.py
[2024-01-15T15:41:36.671+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: latest_only_with_trigger>
[2024-01-15T15:41:36.672+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_short_circuit_decorator.py
[2024-01-15T15:41:36.675+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_short_circuit_decorator>
[2024-01-15T15:41:36.676+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial.py
[2024-01-15T15:41:36.677+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial>
[2024-01-15T15:41:36.677+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial_dag.py
[2024-01-15T15:41:36.678+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial_dag>
[2024-01-15T15:41:36.679+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_time_delta_sensor_async.py
[2024-01-15T15:41:36.679+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_time_delta_sensor_async>
[2024-01-15T15:41:36.680+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_xcomargs.py
[2024-01-15T15:41:36.682+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_xcom_args_with_operators>
[2024-01-15T15:41:36.683+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_xcom_args>
[2024-01-15T15:41:36.683+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_python_dop_operator_3.py
[2024-01-15T15:41:36.684+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.686+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_dop_operator_v3>
[2024-01-15T15:41:36.686+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_latest_only.py
[2024-01-15T15:41:36.687+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: latest_only>
[2024-01-15T15:41:36.688+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_day_of_week_operator.py
[2024-01-15T15:41:36.689+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.690+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.691+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_weekday_branch_operator>
[2024-01-15T15:41:36.691+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_python_decorator.py
[2024-01-15T15:41:36.695+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_python_decorator>
[2024-01-15T15:41:36.695+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_dynamic_task_mapping.py
[2024-01-15T15:41:36.697+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_dynamic_task_mapping>
[2024-01-15T15:41:36.697+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_skip_dag.py
[2024-01-15T15:41:36.699+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_skip_dag>
[2024-01-15T15:41:36.699+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial_objectstorage.py
[2024-01-15T15:41:36.760+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial_objectstorage>
[2024-01-15T15:41:36.760+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_setup_teardown.py
[2024-01-15T15:41:36.762+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_setup_teardown>
[2024-01-15T15:41:36.762+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_branch_datetime_operator.py
[2024-01-15T15:41:36.763+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.764+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.766+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.767+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.768+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_datetime_operator>
[2024-01-15T15:41:36.768+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.768+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_datetime_operator_3>
[2024-01-15T15:41:36.769+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.769+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_branch_datetime_operator_2>
[2024-01-15T15:41:36.770+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_nested_branch_dag.py
[2024-01-15T15:41:36.770+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.774+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.774+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_nested_branch_dag>
[2024-01-15T15:41:36.775+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_trigger_controller_dag.py
[2024-01-15T15:41:36.776+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_trigger_controller_dag>
[2024-01-15T15:41:36.777+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_bash_operator.py
[2024-01-15T15:41:36.777+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.781+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_bash_operator>
[2024-01-15T15:41:36.782+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_bash_operator>
[2024-01-15T15:41:36.782+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_sensor_decorator.py
[2024-01-15T15:41:36.783+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_sensor_decorator>
[2024-01-15T15:41:36.784+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_dag_decorator.py
[2024-01-15T15:41:36.859+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_dag_decorator>
[2024-01-15T15:41:36.859+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial_taskflow_api_virtualenv.py
[2024-01-15T15:41:36.861+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial_taskflow_api_virtualenv>
[2024-01-15T15:41:36.862+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_passing_params_via_test_command.py
[2024-01-15T15:41:36.862+0000] {GetText.py:65} DEBUG - /home/airflow/.local/lib/python3.11/site-packages/cron_descriptor/locale/en_US.mo Loaded
[2024-01-15T15:41:36.865+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_passing_params_via_test_command>
[2024-01-15T15:41:36.865+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_task_group.py
[2024-01-15T15:41:36.867+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_task_group>
[2024-01-15T15:41:36.867+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_params_trigger_ui.py
[2024-01-15T15:41:36.872+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_params_trigger_ui>
[2024-01-15T15:41:36.872+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/example_setup_teardown_taskflow.py
[2024-01-15T15:41:36.876+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: example_setup_teardown_taskflow>
[2024-01-15T15:41:36.876+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/tutorial_taskflow_api.py
[2024-01-15T15:41:36.878+0000] {dagbag.py:504} DEBUG - Loaded DAG <DAG: tutorial_taskflow_api>
[2024-01-15T15:41:36.878+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/plugins/event_listener.py
[2024-01-15T15:41:36.879+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/plugins/workday.py
[2024-01-15T15:41:37.157+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/plugins/listener_plugin.py
[2024-01-15T15:41:37.159+0000] {dagbag.py:328} DEBUG - Importing /home/airflow/.local/lib/python3.11/site-packages/airflow/example_dags/subdags/subdag.py
[2024-01-15T15:41:37.184+0000] {plugins_manager.py:322} DEBUG - Plugins are already loaded. Skipping.
[2024-01-15T15:41:37.184+0000] {plugins_manager.py:506} DEBUG - Integrate DAG plugins
[2024-01-15T15:41:37.188+0000] {taskinstance.py:980} DEBUG - previous_execution_date was called
Changing /home/airflow/sources/logs/dag_id=example_complex/run_id=manual__2024-01-15T15:40:09.891366+00:00/task_id=delete_entry permission to 509
[2024-01-15T15:41:37.192+0000] {task_command.py:423} INFO - Running <TaskInstance: example_complex.delete_entry manual__2024-01-15T15:40:09.891366+00:00 [queued]> on host kd-073023-bdf8b54b-ad46159c-deployment-dd6d9d4d6-85nxx
[2024-01-15T15:41:37.193+0000] {settings.py:371} DEBUG - Disposing DB connection pool (PID 18061)
[2024-01-15T15:41:37.193+0000] {settings.py:234} DEBUG - Setting up DB connection pool (PID 18061)
[2024-01-15T15:41:37.193+0000] {settings.py:303} DEBUG - settings.prepare_engine_args(): Using NullPool
[2024-01-15T15:41:38.859+0000] {cli_action_loggers.py:85} DEBUG - Calling callbacks: []
[2024-01-15T15:41:38.860+0000] {settings.py:371} DEBUG - Disposing DB connection pool (PID 18061)

Task example_complex,delete_entry,manual__2024-01-15T15:40:09.891366+00:00,1,-1 completed successfully

As it can be seen from some steps to other it takes a couple of seconds (not referring between tasks), I mean to the logs from a given task only