ua-snap / prefect

1 stars 0 forks source link

Remove hard coded values from indicator functions #3

Closed BobTorgerson closed 7 months ago

BobTorgerson commented 8 months ago

https://github.com/ua-snap/prefect/blob/4a381b5e107e7cbb5370eadf8caa5fb9592060b1/indicators/indicator_functions.py#L126-L128

Joshdpaul commented 8 months ago

@BobTorgerson one idea I had was to only supply the working directory as a parameter to the Prefect flow, and build all other paths (for {output_dir} {slurm_script} {qc_script}) off of that one input. This would cut down the parameter tweaks in the Prefect dashboard and just provide the trunk of the tree off which all paths are built. The conda_init_script path could also be built here from the working directory, and served into the functions in indicators/indicator_functions.py .

In indicators/generate_indicators.py this might look something like:

if __name__ == "__main__":
    ssh_username = "snapdata"
    ssh_private_key_path = "/home/{ssh_username}/.ssh/id_rsa"
    branch_name = "main"
    working_directory = Path(f"/import/beegfs/CMIP6/{ssh_username}/indicators")
    input_dir = "/import/beegfs/CMIP6/arctic-cmip6/regrid/"
    indicators = "rx1day"
    models = "CESM2 GFDL-ESM4 TaiESM1"
    scenarios = "historical ssp126 ssp245 ssp370 ssp585"

    generate_indicators.serve(
        name="generate_indicators",
        tags=["CMIP6 Indicators"],
        parameters={
            "ssh_username": ssh_username,
            "ssh_private_key_path": ssh_private_key_path,
            "branch_name": branch_name,
            "working_directory": working_directory,
            "indicators": indicators,
            "models": models,
            "scenarios": scenarios,
            "slurm_script": working_directory.joinpath("cmip6-utils/indicators/slurm.py"),
            "qc_script": working_directory.joinpath("cmip6-utils/indicators/qc.py"),
            "conda_init_script": working_directory.joinpath("cmip6-utils/indicators/conda_init.sh"),
            "input_dir": input_dir,
            "output_dir": working_directory.joinpath("output/"),
        },
    )
Joshdpaul commented 7 months ago

Already done!