Introduces build_dynamic_tenant_year_taskgroup, for running dynamic task mapping in earthbeam. This allows you to run multiple instances of earthmover, etc. if you receive multiple files
Some things that definitely need review:
dynamicPythonOperator. I added this to allow for some kwargs to be static, some dynamic. But there is likely a better way to accomplish this
evaluate_kwargsand handling of kwargs in general. This probably overlaps with Jay's work to make a configurable factory.. we need to be flexible to multiple different structures of earthmover kwargs, depending on the configured template. I sort of have that here, but the context from here is hard coded to include one input_file, when we likely need to be flexible to multiple input parameters and other earthmover kwarg constructions??
Introduces build_dynamic_tenant_year_taskgroup, for running dynamic task mapping in earthbeam. This allows you to run multiple instances of earthmover, etc. if you receive multiple files
Some things that definitely need review:
dynamicPythonOperator. I added this to allow for some kwargs to be static, some dynamic. But there is likely a better way to accomplish thisevaluate_kwargsand handling of kwargs in general. This probably overlaps with Jay's work to make a configurable factory.. we need to be flexible to multiple different structures of earthmover kwargs, depending on the configured template. I sort of have that here, but thecontextfrom here is hard coded to include one input_file, when we likely need to be flexible to multiple input parameters and other earthmover kwarg constructions??I have been testing with this refactored DAG in texas - https://github.com/edanalytics/stadium_txexchange/blob/feature/refactor_dynamic_earthbeam/airflow/dags/refactor_dynamic_earthbeam_dags.py