Open rdeveloper21 opened 2 months ago
boring-cyborg[bot]
commented
2 months ago Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
RNHTTR
commented
2 months ago Are you able to try to reproduce on the latest version of the celery provider (3.6.2)?
NBardelot
commented
2 months ago Wouldn't the amqps:// protocol only make TLS-enabled connections? It seems AIRFLOW__CELERY__SSL_ACTIVE must then be True in this case and Airflow behaviour is correct.
See https://www.rabbitmq.com/docs/uri-spec#the-amqps-uri-scheme
The "amqps" URI scheme is used to instruct a client to make an secured connection to the server.
And disabling TLS would then go with switching back to the amqp:// URI scheme.
rdeveloper21
commented
2 months ago @NBardelot My mistake... actually in the above steps to reproduce there is one correction... Implement SSL for rabbitmq broker. Update below variables: "AIRFLOWCELERYBROKER_URL": "amqps://user:password@ip:port/" "AIRFLOWCELERYSSL_ACTIVE": "True" "AIRFLOWCELERYSSL_CACERT": "/path/to/cacert.pem" and try to connect broker.
In stead of False it is True.
rdeveloper21
commented
2 months ago @RNHTTR We install all the required packages using below command: pip install 'apache-airflow[celery]==2.7.2' --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.8.txt"
Tried with celery provider (3.6.2). Observing same error. Below is Traceback:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/Admin/miniconda3/envs/airflow/bin/airflow", line 8, in
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.7.2
What happened?
Basically in our airflow setup we had implemented rabbitmq broker url connection for celery executor. Later on, due to certain vulnerabilities issue, we had implemented SSL for rabbitmq connection. To handle same we changed the value of below config variables as per documentation: "AIRFLOWCELERYBROKER_URL": "amqps://user:password@ip:port/" "AIRFLOWCELERYSSL_ACTIVE": "True" "AIRFLOWCELERYSSL_CACERT": "/path/to/cacert.pem"
But still we were observing airflow exception: raise AirflowException( airflow.exceptions.AirflowException: Exception: There was an unknown Celery SSL Error. Please ensure you want to use SSL and/or have all necessary certs and key (The broker you configured does not support SSL_ACTIVE to be True. Please use RabbitMQ or Redis if you would like to use SSL for broker.).
Then after going through source code we found below:
Due to the condition >>"amqp://" in broker_url << the exception was getting raised. When tried by updating the condition and also commenting the "keyfile" and "certfile" in broker_use_ssl, we were able to connect to rabbitmq and proceed.
What you think should happen instead?
The condition when 'celery_ssl_active' is True should be revisited and updated. As the condition if "amqp://" in broker_url is not justified.
How to reproduce
Implement SSL for rabbitmq broker. Update below variables: "AIRFLOWCELERYBROKER_URL": "amqps://user:password@ip:port/" "AIRFLOWCELERYSSL_ACTIVE": "True" "AIRFLOWCELERYSSL_CACERT": "/path/to/cacert.pem" and try to connect broker.
Operating System
linux
Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.3.4
Deployment
Other Docker-based deployment
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct