apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.55k stars 14.16k forks source link

ObjectStoragePath should be template-able #40361

Closed fritz-astronomer closed 2 months ago

fritz-astronomer commented 3 months ago

Description

ObjectStoragePath should be able to handle Jinja Templates

Use case/motivation

If I have an operator, that has an ObjectStoragePath argument

class MyOperator(BaseOperator):
  template_fields = ('path',)

  def __init__(path, **kwargs):
    self.path = path
    super().__init__(**kwargs)

and am using it in a DAG, with templating

MyOperator(
  ...,
  path=ObjectStoragePath(
    "s3://my_bucket/foo/{{ ds_nodash }}/file.csv",
    conn_id="s3"
  )
)

Templating didn't seem to work

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

fritz-astronomer commented 3 months ago

acc'd to this https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#jinja-templating it looks like ObjectStoragePath doesn't have template_fields =

I tried a few things:

ObjectStoragePath.template_fields = (...) doesn't work, because the init field is *args (I tried with args)

I can't do render_template(...) in my operator, I think, because ObjectStoragePath is read-only

gyli commented 2 months ago

The current render_template does not handle the case when value is ObjectStoragePath, and I think it can be added.

As a workaround before it's added, I think you can override render_template in your operator to render and rebuild the path.