search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.13k stars 14.31k forks source link

Add `dag_id` and `map_index` to the default template variables #40958

Open pedro-cf opened 3 months ago

pedro-cf commented 3 months ago

Description

Add dag_id and map_index as default variables to the template variables (accessible via task parameters).

Use case/motivation

Generally useful and specifically easier access to these variables within operators that have serialization issues with the dag_run and ti objects, such as the PythonVirtualEnvOperator.

Related issues

Are you willing to submit a PR?

Code of Conduct

shahar1 commented 3 months ago
  1. Is it a feature suggestion, or a suggestion for improving the documentation? Please clarify the description - it would also be useful to provide some code examples that will explain the motivation.
  2. Is this PR a recreation of #40859? Why was the latter closed?
pedro-cf commented 3 months ago
  1. Is it a feature suggestion, or a suggestion for improving the documentation?

Please clarify the description - it would also be useful to provide some code examples that will explain the motivation.

  1. Is this PR a recreation of #40859? Why was the latter closed?
  1. Feature Request/Suggestion. I believe I give a solid reason:

Generally useful and specifically easier access to these variables within operators that have serialization issues with the dag_run and ti objects, such as the PythonVirtualEnvOperator.

  1. Different variables in this issue.
shahar1 commented 3 months ago
  1. Is it a feature suggestion, or a suggestion for improving the documentation?

Please clarify the description - it would also be useful to provide some code examples that will explain the motivation.

  1. Is this PR a recreation of Add 'dag_id' and 'dag_run_id' to the default template variables #40859? Why was the latter closed?
  1. Feature Request/Suggestion. I believe I give a solid reason:

Generally useful and specifically easier access to these variables within operators that have serialization issues with the dag_run and ti objects, such as the PythonVirtualEnvOperator.

  1. Different variables in this issue.

There's currently an open PR (#41039) for solving #34158. When this PR is merged, will there still be a justification for the suggested feature?

pedro-cf commented 3 months ago

There's currently an open PR (#41039) for solving #34158. When this PR is merged, will there still be a justification for the suggested feature?

I believe so, these are key variables, more important than some of the variables already available. It's also a much cleaner way to access these directly.

uranusjr commented 3 months ago

I wonder if we should try to add some more, and remove ti and dag_run altogether. Exposing those objects directly to the user is kind of problematic.

Or, maybe we should make those a proxy instead of the actual object, and remove things like task_id and run_id.

uranusjr commented 3 months ago

Adding this to 3.0 since I think we need something to track the discussion. I’m not particularly in favour or disfavour to the exact solution proposed here.