mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
MIT License
2.07k stars 100 forks source link

passing/overriding config via `flask ...run <param>` #85

Open leo-schick opened 2 years ago

leo-schick commented 2 years ago

I would wish an option in the cli command flask mara_pipelines.ui.run to override variables.

Example:

I have a app.config.py file:

from enum import Enum

class ProcessingMode(Enum):
    FULL = 'full'
    INCREMENTAL = 'incremental'

def processing_mode() -> ProcessingMode:
    """The processing mode to be used"""
    return ProcessingMode.INCREMENTAL

def default_window_in_days() -> int:
    """The default refresh window for models when running a incremental sync."""
    return 90

Now I would like to override the config for a singe pipeline execution.

Sample design:

# sets the processing_mode to FULL
flask mara_pipelines.ui.run --patch-config app.config.processing_mode=FULL

# sets the default_window_in_days to 180 days
flask mara_pipelines.ui.run --patch-config app.config.default_window_in_days=FULL

The parameter name should maybe not be called --patch-config <str>. Terraform uses e.g. -var=<...>/-var-file=<...>