search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.43k stars 14.36k forks source link

Add DeltaSharing provider and sensor. #19473

Closed dmoore247 closed 1 year ago

dmoore247 commented 3 years ago

Description

Using this Issue to add initial support for a DeltaSharing provider. We would clone an existing provider and modify to monitor for changes in delta share tables via the rest protocol. This is a lightweight mechanism to pause before the entire Delta Share update is processed.

Use case/motivation

Use Case: As a user of Airflow and DeltaSharing to optimally process new data refreshes, I'd like to wait on a new update on a specified Delta Sharing table before progressing the rest of my DAG.

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

boring-cyborg[bot] commented 3 years ago

Thanks for opening your first issue here! Be sure to follow the issue template!

eladkal commented 2 years ago

@dmoore247 are you still working on this issue?

alexott commented 2 years ago

I can pickup this work, starting with sensor (although maybe it makes sense to have a separate issue for a sensor)

eladkal commented 2 years ago

No need for multiple issues :) Issues are not really required. You can open PRs freely.

eladkal commented 1 year ago

Policy for adding new provider is explained at https://github.com/apache/airflow/blob/main/PROVIDERS.rst

I'm going to close this issue as the process of adding a provider require investment and commitments from the requester thus it's unlikely that someone will just pick up this request and implement it so no reason to keep this issue open.