search

data-as-code / dac

Python Data as Code core implementation
MIT License
6 stars 0 forks source link

Support `load.py` and `schema.py` as templates with values injected during `dac pack` #11

Open francesco086 opened 1 year ago

francesco086 commented 1 year ago

As a dac package producer I would like to have the possibility to insert parameters in the load.py and schema.py files that will be filled when running dac pack so that I can re-use the same templates

Example

import dask.dataframe as dd

def load() -> dd.DataFrame:
    return dd.read_parquet(path="az://STORAGE_CONTAINER_NAME/DATA_FOLDER/*.parquet", 
                           storage_options={"account_name": "STORAGE_ACCOUNT_NAME", "anon": False})

Here, STORAGE_ACCOUNT_NAME, STORAGE_CONTAINER_NAME, and DATA_FOLDER could be parametrized to allow re-usability