astronomer / astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
https://astro-sdk-python.rtfd.io/
Apache License 2.0
335 stars 40 forks source link

Easier access to the Airflow context #812

Open rossturk opened 1 year ago

rossturk commented 1 year ago

Please describe the feature you'd like to see

For getting Airflow context, get_current_context() is supported, and it seems to work fine. However, I had to search through the Airflow docs to find it and the resulting code is a bit wordy:

@aql.dataframe
def do_stuff():
    context = get_current_context()
    data_interval_start = context["data_interval_start"]
    data_interval_end = context["data_interval_end"]
      ...

I think this is something that pipeline builders will need to do on every pipeline that can backfill, which should be a lot of 'em.

Describe the solution you'd like

Could context be available via kwargs like they are in TaskFlow?

@aql.dataframe
def do_stuff(data_interval_start=None,data_interval_end=None):
    ...

Are there any alternatives to this feature?

We could document get_current_context() as the correct way to do this in the Astro SDK docs.

Acceptance Criteria

utkarsharma2 commented 1 year ago

@feluelle ^^

utkarsharma2 commented 1 year ago

@rossturk Thanks for creating this ticket, we'll take a look at it. :)

phanikumv commented 1 year ago

@kaxil any thoughts on how critical this is ? Considering the backlog we have at the moment, I dont think we can work on this atleast till next sprint

kaxil commented 1 year ago

@phanikumv - low

tatiana commented 1 year ago

As agreed with @jlaneve, we're deferring this to the milestone 1.7.