fabricks-framework / fabricks

MIT License
5 stars 0 forks source link

Extenders should take parameters #29

Open aersam opened 6 days ago

aersam commented 6 days ago

we currently have 3 different unpivot extenders, which could be reduced to one if we had parameter support. I propose to have something like this:


@extender(name="unpivot")
def unpivot(df: DataFrame, parameters: dict) -> DataFrame:
    prefix = parameters.get("untouched_prefix", "val_")
    untouched_columns = [f"`{c}`" for c in df.columns if c.startswith("__") or not c.lower().startswith(prefix)]
    val_columns = [f"`{c}`" for c in df.columns if c.lower().startswith(prefix)]
    return df.unpivot(
        untouched_columns,
        val_columns,
        variableColumnName=parameters.get("variableColumnName", "date_name"),
        valueColumnName=parameters.get("valueColumnName", "date_value"),
    )
aersam commented 6 days ago

I thought about also passing the job config, but this might lead to bad things