search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
35.3k stars 13.79k forks source link

Improve SFTPOperator with directory transfer and DELETE operation #40365

Open Dawnpool opened 2 weeks ago

Dawnpool commented 2 weeks ago

Description

Currently, the SFTPOperator in airflow providers does not support directory transfers. You have to specify every filename in a folder if you want to transfer the whole folder. Additionally, the operator supports only PUT and GET methods, not DELETE methods.

I think this operator would be more powerful if it could transfer an entire folder by specifying just the folder name as well as delete files and folders with DELETE method.

Use case/motivation

I want to copy a folder to an SFTP remote server using only the folder name. Before copying, I want to delete the already existing folder on the SFTP remote server to ensure it is overwritten.

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

boring-cyborg[bot] commented 2 weeks ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.