search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.65k stars 14.18k forks source link

Airflow airflow:2.0.2 UI is not getting updated with dag status #21668

Closed nitra-1 closed 2 years ago

nitra-1 commented 2 years ago

Apache Airflow version

2.0.2

What happened

The dags run and status gets updated but over the period like in 6-8 hours time and randomly the schedulers are seen in queued state. And once we restart the scheduler immediately all the tasks are seen as success with no queued tasks. With this we have to restart the scheduler quite frequently.

What you expected to happen

Expected behaviour that each scheduler should get executed and status to be updated in the web ui

How to reproduce

Install the airflow in master-worker HA mode. Deploy some schedulers and observe the status of the schedulers

Operating System

VERSION="18.04.5 LTS (Bionic Beaver)"

Versions of Apache Airflow Providers

apache-airflow-providers-airbyte==1.0.0 apache-airflow-providers-amazon==1.4.0 apache-airflow-providers-apache-beam==2.0.0 apache-airflow-providers-apache-cassandra==1.0.1 apache-airflow-providers-apache-druid==1.1.0 apache-airflow-providers-apache-hdfs==1.0.1 apache-airflow-providers-apache-hive==1.0.3 apache-airflow-providers-apache-kylin==1.0.1 apache-airflow-providers-apache-livy==1.1.0 apache-airflow-providers-apache-pig==1.0.1 apache-airflow-providers-apache-pinot==1.0.1 apache-airflow-providers-apache-spark==1.0.3 apache-airflow-providers-apache-sqoop==1.0.1 apache-airflow-providers-celery==1.0.1 apache-airflow-providers-cloudant==1.0.1 apache-airflow-providers-cncf-kubernetes==1.2.0 apache-airflow-providers-databricks==1.0.1 apache-airflow-providers-datadog==1.0.1 apache-airflow-providers-dingding==1.0.2 apache-airflow-providers-discord==1.0.1 apache-airflow-providers-docker==1.2.0 apache-airflow-providers-elasticsearch==1.0.4 apache-airflow-providers-exasol==1.1.1 apache-airflow-providers-facebook==1.1.0 apache-airflow-providers-ftp==1.1.0 apache-airflow-providers-google==3.0.0 apache-airflow-providers-grpc==1.1.0 apache-airflow-providers-hashicorp==1.0.2 apache-airflow-providers-http==1.1.1 apache-airflow-providers-imap==1.0.1 apache-airflow-providers-jdbc==1.0.1 apache-airflow-providers-jenkins==1.1.0 apache-airflow-providers-jira==1.0.2 apache-airflow-providers-microsoft-azure==2.0.0 apache-airflow-providers-microsoft-mssql==1.1.0 apache-airflow-providers-microsoft-winrm==1.2.0 apache-airflow-providers-mongo==1.0.1 apache-airflow-providers-mysql==1.1.0 apache-airflow-providers-neo4j==1.0.1 apache-airflow-providers-odbc==1.0.1 apache-airflow-providers-openfaas==1.1.1 apache-airflow-providers-opsgenie==1.0.2 apache-airflow-providers-oracle==1.1.0 apache-airflow-providers-pagerduty==1.0.1 apache-airflow-providers-papermill==1.0.2 apache-airflow-providers-plexus==1.0.1 apache-airflow-providers-postgres==1.0.2 apache-airflow-providers-presto==1.0.2 apache-airflow-providers-qubole==1.0.2 apache-airflow-providers-redis==1.0.1 apache-airflow-providers-salesforce==2.0.0 apache-airflow-providers-samba==1.0.1 apache-airflow-providers-segment==1.0.1 apache-airflow-providers-sendgrid==1.0.2 apache-airflow-providers-sftp==1.2.0 apache-airflow-providers-singularity==1.1.0 apache-airflow-providers-slack==3.0.0 apache-airflow-providers-snowflake==1.3.0 apache-airflow-providers-sqlite==1.0.2 apache-airflow-providers-ssh==1.3.0 apache-airflow-providers-tableau==1.0.0 apache-airflow-providers-telegram==1.0.2 apache-airflow-providers-trino==1.0.0 apache-airflow-providers-vertica==1.0.1 apache-airflow-providers-yandex==1.0.1 apache-airflow-providers-zendesk==1.0.1

Deployment

Other

Deployment details

Installation done using

pip3 install apache-airflow[all]

Anything else

no

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 2 years ago

Thanks for opening your first issue here! Be sure to follow the issue template!

nitra-1 commented 2 years ago

noname As seen in the image, schedulers remain in the scheduled stated, even if there is no scheduler running these scheduler remain in scheduled state. And once i restart the scheduler these all will be gone from scheduled state to success

nitra-1 commented 2 years ago

Any update on this issue ?

potiuk commented 2 years ago

I think you should simply upgrade to newer version and let us know if the issue is fixed. There were a number of problems fixed since 2.0.2.

Anyway - this is your only option because there won't be any more 2.0. 2.1. and soon also 2.2.* releases. The way to apply bugfixes from newer version is simply to ... update to the newer versions.

There is no guarantee that the issue will be fixed there, but there is a good chance it will.

I will convert it to a discussion - please upgrade and let us know if this fixed the issue.