search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
35.95k stars 13.97k forks source link

glob syntax can be similar to .gitignore in .airflowignore files #40672

Open YuvaPhani27 opened 1 month ago

YuvaPhani27 commented 1 month ago

Description

Extend glob pattern matching support similar to .gitignore in .airflowignore

Use case/motivation

I recently upgraded Airflow from version 2.6.2 to 2.9.2 and noticed a change in version 2.6.3 related to .airflowignore handling. My requirement is to exclude everything except certain folders within the dags_folder path. Previously, I achieved this using pattern ^(?!(foo|bar)).* with the default dag_ignore_file_syntax = regexp. However, after the upgrade, this approach stopped working because Google's RE2 engine doesn't seem to support (?!) as expected. I attempted switching dag_ignore_file_syntax = glob and observed that the current glob support is very limited. Hence requesting this feature to provide similar functionality as of .gitignore where I can define the pattern like !{foo/*,bar/*}.

Related issues

NA

Are you willing to submit a PR?

Code of Conduct

boring-cyborg[bot] commented 1 month ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

poorvirohidekar commented 1 month ago

@potiuk This seems to be interesting and I would like to try this with some assistance. Could you please assign it to me?

potiuk commented 1 month ago

@potiuk This seems to be interesting and I would like to try this with some assistance. Could you please assign it to me?

Just to set an expectation - there will not be much of an assistance I am afraid, we rarely do "hand-holding" for these kind of issues, other than reviews of the PR. but this one does not have much of specific "airflow" knowledge to get - it's pretty general programming task and you should be able to find your ways there I think, so let me assign it to you.