search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.4k stars 3.97k forks source link

Source File: dynamically grab files with dates in the names #12687

Open kylejbrk opened 2 years ago

kylejbrk commented 2 years ago

Tell us about the problem you're trying to solve

If this isn't currently possible (didn't see it in the docs), it would be nice to have a feature where you can grab files that have dates in the names.

Describe the solution you’d like

For example example if I have a daily file that is delivered on an SFTP that will always include the current date, the airbyte operator should be able to grab it. Example file name: "daily_file_2022-05-07.csv"

Describe the alternative you’ve considered or used

I currently use airflow to do this which uses the DAG execution date.

alonsomedo commented 1 year ago

Could you bring an example of how you are doing this from airflow? because the airbyte operator doesn't allow sending any params for that.