Create task specific modules for all jobs in the workflow.

[aerorahul]

Description A spreadsheet has been created here to collect the information to proceed on this task. This spreadsheet currently contains task list for WCOSS2. Orion will be added shortly.

Some preliminary work was done to read the spreadsheet (as a CSV) and collect this information. It can be found in this branch. Look for ush/create_modules.py.

Requirements The code should be able to create lua modulefiles for each job in the workflow based on inputs from a YAML file maintained for that machine. The YAML file should contain a list of all the jobs, the name of its modules and modulepaths. module_version may be obtained from the environment maintained in run.ver.<platform>. Order of the modules listed in the yaml may be important to be maintained. The YAML file may define environment variables to be included in the modulefile. The structure of the YAML file may be updated as the development progresses.

Acceptance Criteria (Definition of Done) The outcome is to generate lua modulefiles for each job per machine.

(Optional): Suggest A Solution Example wcoss2.yaml:

common:
  modulepaths:
    - None
  modules:
    - PrgEnv-intel/${PrgEnv_intel_ver}
    - craype/${craype_ver}
    - intel/${intel_ver}
    - prod_util/${prod_util_ver}
eobs:
  help: Load environment to run eobs job on WCOSS2
  whatis: eobs run environment
  modules:
    - cray-mpich/${cray_mpich_ver}
    - cray-pals/${cray_pals_ver}
    - python/${python_ver}
    - hdf5/${hdf5_ver}
    - netcdf/${netcdf_ver}
    - crtm/${crtm_ver}
prep:
  help: Load environment to run prep job on WCOSS2
  whatis: prep run environment
  modules:
    - hdf5/${hdf5_ver}
    - netcdf/${netcdf_ver}
  extra:
    modulepaths:
      - /lfs/h2/emc/global/save/emc.global/git/prepobs/module
    modules:
      - prepobs/${prepobs_ver}

Output wcoss2/prep.lua modulefile:

help([[
Load environment to run prep job on WCOSS2
]])

load(pathJoin("PrgEnv-intel", os.getenv("PrgEnv_intel_ver") or "8.1.0"))
load(pathJoin("craype", os.getenv("craype_ver") or "2.7.13"))
load(pathJoin("intel", os.getenv("inter_ver") or "19.1.3.304"))
load(pathJoin("prod_util", os.getenv("prod_util_ver") or "2.0.9"))

load(pathJoin("hdf5", os.getenv("hdf5_ver") or "1.10.6"))
load(pathJoin("netcdf", os.getenv("netcdf_ver") or "4.7.4"))

prepend_path("MODULEPATH", pathJoin("/lfs/h2/emc/global/save/emc.global/git/prepobs/module"))
load(pathJoin("prepobs", os.getenv("prepobs_ver") or "1.0.1"))

whatis("Description: prep run environment")

[ryanlong1004]

Joining.

[ryanlong1004]

@aerorahul Is it 'legal' to specify a module without a version as below? load(pathJoin("rocoto"))

[aerorahul]

No. A version is required.

[ryanlong1004]

@aerorahul See below example. I think this is prob the most intuitive way. Each '---' represents a separate .lua file to be exported.

The python would check that the [modulepaths, modules, environment, help, whatis] are valid and kick back anything not in that list.

Each document heading (common, eobs, etc) would be used as the lua output file name so common -> common.lua.

The python itself would lookup the the env variables and substitute them in its serialization to Lua.

---
common:
  modulepaths:
    - None
  modules:
    - PrgEnv-intel/$PrgEnv_intel_ver
    - craype/$craype_ver
    - intel/$intel_ver
    - prod_util/$prod_util_ver
  environment:
    - WGRIB2: wgrib2
  help: >
    Load environment to run prep job on WCOSS2
  whatis: >
    An example of what is.
---
eobs:
  modules:
    - cray-mpich/$cray_mpich_ver
    - cray-pals/$cray_pals_ver
    - python/$python_ver
    - hdf5/$hdf5_ver
    - netcdf/$netcdf_ver
    - crtm/$crtm_ver
---
prep:
  modules:
    - hdf5/$hdf5_ver
    - netcdf/$netcdf_ver
  extra:
    modulepaths:
      - /lfs/h2/emc/global/save/emc.global/git/prepobs/module
    modules:
      - prepobs/$prepobs_ver

[aerorahul]

I like the use of --- as a file-separator. Though, I don't like a separate common.lua. The job.lua file should be complete i.e. it should have the parts of common and extra (if extra is a part of the job).

This is so that if an user were to test a different module in common for a particular job, they can specify the keys from common in the job and the job will take precedence.