search

bidhya / verse

0 stars 0 forks source link

readme

Blender

The notes here are now only specific to LIS related 1km runs.
010 refers to 0.01 degree (1km) resolution. Future runs can thus be readily adapted to any higher/lower resolution as required.

Project folder: /discover/nobackup/projects/coressd

Subfolder

OSU: Modis snow cover data downloaded here.
mwrzesie: LIS data generated by Melisa
Github: codes related to Input generation, Blender runs, and post-processing.

  1. verse : scripts (both Python and Julia)
  2. Slurm_Blender : slrum jobs related to generation of input files and post-processing of Blender runs.
  3. slurm_jobs : will be created by blender run automatically as required.
  4. Blender_notebooks: Jupyter notebooks. Mostly for prototyping and analysis of data. But two imporant ones for Blender workflow:
    • LIS/Extract_LIS_1km.ipynb : used to process LIS data. Uses papermill to run as a slurm job. Plan is to convert it to python script when everything is tested and finalized.
    • Modis/Forest/05_MOD44B_on_Discover.ipynb : use directly from Jupyterhub on discover. used to process annual tree-cover data. Since, data already processed no need to run again.
    • paper_outputs : not required to check per se. Output notebooks created by papermill is saved here.
    • rest of notebooks were created during various phases of project and mostly used for protyping of ideas and analysis.

Blender: Blender related data processed by various scripts.

  1. Inputs_010/lis: LIS and MODIS files staged by water year. Input files for Blender run.
  2. Modis: MODIS intermediate files only
    • MOD44B : modis tree-cover reprojected and matched to LIS data
    • MOD10A1F : modis snow-cover data corrected for tree and clipped to match LIS data
      Runs: Blender runs saved relative to resolution. 010 => 1km
    • 010: 1km (1 degree) runs using LIS data staged by WaterYear
    • 010/WY2015/temp_nc: temporary netcdf files for each slice or run
    • 010/WY2015/outputs: Final PAN Americal Outputs created by merging files from "temp_nc" folder

Order of Script Execution for coressd project =============================================

I: To Prepare data

  1. a_lis_process.sh : Process LIS data using papermill/jupyter notebook combination
    Use multiple cores
    Outputs: All daily LIS netcdf files saved separately as follows ../Blender/Inputs_010/lis/WY2015

    • Snowf_tavg.nc
    • SWE_tavg.nc
    • Tair_f_tavg.nc
    • Qg_tavg.nc
    • SCF.nc (will be created by next slurm job (b_process_modis_cgf.sh))

  2. b_process_modis_cgf.sh: Extract NA (North America) scale daily Modis_CGF matching the LIS rasters

    • uses process_modis_cgf.py
    • uses ../OSU/MOD10A1F.061/MODIS_Proc/download_snow/.. for snow-cover data
    • uses MOD44B for tree-cover correction
    • Hard dependency to use "Qg_tavg.nc" as template and mask
    • intermediate outputs: ../coressd/Blender/Modis/MOD10A1F/clipped2015_010: clipped and matched to seup resolution
    • ../Blender/Inputs_010/lis/WY2015/SCF.nc : ie, same location as LIS outputs as above.

II: To Run Blender Julia Code

================================

  1. On HPC (Discover): the following julia script will generate slurm jobs for a water_year.

    • cd to Github folder
    • julia verse/Julia/submit_slurm.jl 2015 2 010 # example: for WY2015 with resolution=010 and adaptive slice = 2 rows.
    • generate ~600 slurm jobs that calls the following julia script:
    • julia verse/Julia/call_Blender_v18.jl WY2010 1 17 010 : example script for Blender run for WY2010 from slice 1 to 17
    • slurm job folder:
      • ../slurm_jobs/010/2015/
        • .out/ : slurm outputs saved here
          Blender Outputs: ../Blender/Runs/010/WY2015/temp_nc/ : These netcdf files for each slice will be combined by next script, after which this folder can be deleted.
    • Aside: locally for small area: use "test" prefix to WY. Example: test_WY2015 while calling call_Blender julia script.
    • julia verse/Julia/call_Blender_v18.jl test_WY2015 3005 3006 010

  2. Post-processing (Required for continental map): Combine individual nc_files into PAN American netcdf file

    • uses: Github/Slurm_Blender/d_combine_out_nc_files.sh
    • calls: julia verse/Julia/combine_nc_files.jl WY$water_year $RES