search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.15k stars 14.04k forks source link

Add a compatibility with Opensearch search for log management #33619

Open djadeau opened 1 year ago

djadeau commented 1 year ago

Description

Dear Airflow team,

Today, it is possible to configure Airflow to read logs of the tasks from ElasticSearch.

Is it possible to add the support of Opensearch as source of the task logs ?

Thank you.

Use case/motivation

Log sinks are often used in large companies. Concentrate logs in a log sinks is very practical for searches.

Opensearch and Elasticsearch have similar methods, so it should not create major change.

ElasticSearch has added some checks in the client library, so it is not possible anymore to Opensearch with the elasticsearch library.

Cloud providers such as OVHCloud or Scaleway, propose Opensearch as log manager as a service.

Opensearch is a full opensource solution contrary to Elasticsearch.

Related issues

No

Are you willing to submit a PR?

Code of Conduct

boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

eladkal commented 1 year ago

@cjames23 is working on opensearch hook

cjames23 commented 11 months ago

Sorry about the delay on this. If no one has made any of the Open Search capabilities I am almost complete with the hook and some cursory operators for creating indexes, documents and searching an index. I need to finish writing unit and system tests and documentation.

cjames23 commented 11 months ago

@eladkal do you want to assign this to me with the two open PRs I have.

cjames23 commented 10 months ago

OpenSearch provider was merged but does not have log integration yet for airflow task log integrations. Currently working on adding support for the AWS Managed version of OpenSearch and then I will begin working on the log integrations.

erikvatt commented 6 months ago

@cjames23 Have you started working on OpenSearch log integration?

eladkal commented 6 months ago

I think that https://github.com/apache/airflow/pull/34693 has most of the needed logic for AWS integration it just require adjustments after the OpenSearch provider was implemented. Once this is completed we can start work on log integration. Not sure though if @cjames23 is working on it

ftforce commented 5 months ago

Any updates on this?