search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.27k stars 14.34k forks source link

Deployment: Airflow 2.9.1 Installed on Kubernetes via Helm Chart - Opensearch Connection Issue (Error 406) #39718

Open emirhanaydin61 opened 6 months ago

emirhanaydin61 commented 6 months ago

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Hello, we are using airflow 2.4.0 and were reading logs from opensearch. When we upgraded airflow to version 2.9.1, we encountered this error: "elasticsearch.ApiError: ApiError(406, 'Content-Type header [application/vnd.elasticsearch+json; compatible-with=8] is not supported', 'Content-Type header [application/vnd.elasticsearch+json; compatible-with=8] is not supported')". Could you please provide support regarding the solution to this issue?

What you think should happen instead?

No response

How to reproduce

Upgrade airflow from version 2.4.0 to 2.9.1. Read logs from Opensearch. Execute a specific workflow or perform a particular operation. Observe the error message.

Operating System

CentOS Linux 7 (Core) / kubernetes v1.21.5

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow version 2.9.1 deployed on Kubernetes using Helm chart. Opensearch version 2.11.1.

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 6 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

adrienyhuel commented 6 months ago

Hello @emirhanaydin61

It is related to #33619

Airflow use "apache-airflow-providers-elasticsearch" to read logs, and the latests versions aren't compatible with Opensearch.

You can downgrade this python package to version 5.0.0, it is the lastest version which still support Opensearch.

eladkal commented 5 months ago

Yes this will be solved when https://github.com/apache/airflow/issues/33619 is resolved but this reports indicate that this is more than just a feature request as https://github.com/apache/airflow/issues/33619 states as users who are unable to migrate are blocked. ~cc @shubham22 maybe worth considering for AWS to priortize this task. I think it's very close to the finish line as mentioned in https://github.com/apache/airflow/issues/33619#issuecomment-1985675070~ (on second thought this isn't really about the Amazon OpenSearch but about OpenSearch in general)