search

desihub / desimodules

Maintain the default set of DESI product versions and environment modules loaded.
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Environment variables are multiply-defined in several packages including desimodules itself #49

Closed weaverba137 closed 1 month ago

weaverba137 commented 4 months ago

Environment variable audit

Introduction

desimodules loads many packages, some of which define extra environment variables. desimodules itself defines several environment variables.

Finding environment variables in desiconda/desimodules

Find "interesting" setenv commands in all modulefiles:

cd /global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/modulefiles
grep -r setenv * | grep -v toupper

The sections below are ordered by the module load commands in desimodules/24.4 (Jura).

Environment variables that "point to the package" are ignored, e.g. setenv RR_TEMPLATE_DIR $PRODUCT_DIR.

Summary of Conflicts

For each conflict, the order of the list of conflicting packages is the order in which they are loaded by desimodules/24.4.

Although these variables are not in conflict, we may want to think about centralizing where MP variables are defined to prevent future conflicts:

Package Detail

desiconda

    unsetenv PYTHONSTARTUP
    setenv PROJ_LIB /global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/conda/share/proj
    # setenv SLURM_CPU_BIND cores
    setenv CRAYPE_LINK_TYPE dynamic
        setenv LD_PRELOAD $preldlib:$env(LD_PRELOAD)
        setenv LD_PRELOAD $preldlib

desitree

    setenv DESI_MOCKS $root/mocks
    setenv DESI_SPECTRO_DATA $root/spectro/data
    setenv DESI_SPECTRO_REDUX $root/spectro/redux
    setenv DESI_SPECTRO_SIM $root/spectro/sim
    setenv DESI_SURVEYOPS $root/survey/ops/surveyops/trunk
    setenv DESI_TARGET $root/target
    setenv DESI_WWW $root/www
    setenv PIXPROD $env(USER)
    setenv SPECPROD $env(USER)
    setenv PRODNAME $env(USER)

desitarget

    setenv TARG_DIR $env(DESI_ROOT)/target/catalogs
    setenv MTL_DIR  $env(DESI_ROOT)/target/surveyops/mtl
    setenv DUST_DIR $env(DESI_ROOT)/external/dust/v0_1
    setenv CMX_DIR  $env(DESI_ROOT)/target/cmx_files
    setenv SCND_DIR $env(DESI_ROOT)/target/secondary
    setenv MASK_DIR $env(DESI_ROOT)/target/masks
    setenv GAIA_DIR $env(DESI_ROOT)/target/gaia_dr2
    setenv URAT_DIR $env(DESI_ROOT)/target/urat_dr1
    setenv TYCHO_DIR $env(DESI_ROOT)/target/tycho_dr2
    setenv TOO_DIR  $env(DESI_ROOT)/target/ToO
    setenv ZCAT_DIR $env(DESI_ROOT)/spectro/redux/daily
    setenv QN_MODEL_FILE $env(DESI_ROOT)/target/catalogs/lya/qn_models/qn_train_coadd_indtrain_0_0_boss10.h5
    setenv SQ_MODEL_FILE $env(DESI_ROOT)/target/catalogs/lya/sq_models/BOSS_train_64plates_model.json
    setenv SKYBRICKS_DIR $env(DESI_ROOT)/target/skybricks/v3

desispec

desispec/main and desispec/0.63.0 have different values of QN_MODEL_FILE, 0.63.0 is shown here:

    setenv DESI_SPECTRO_CALIB $env(DESI_ROOT)/spectro/desi_spectro_calib/trunk
    setenv DESI_SPECTRO_DARK $env(DESI_ROOT)/spectro/desi_spectro_dark/v2209
    setenv MPICH_GNI_FORK_MODE FULLCOPY
    setenv KMP_AFFINITY disabled
    # setenv QN_MODEL_FILE $env(DESI_ROOT)/target/catalogs/lya/qn_models/qn_train_coadd_indtrain_0_0_boss10.h5
    setenv QN_MODEL_FILE $env(DESI_ROOT)/science/lya/qn_models/desi/qn_train_desi_guadalupe_linear.h5
    setenv SQ_MODEL_FILE $env(DESI_ROOT)/target/catalogs/lya/sq_models/BOSS_train_64plates_model.json

desisim

setenv DESI_BASIS_TEMPLATES $env(DESI_ROOT)/spectro/templates/basis_templates/v3.2

fiberassign

Note this is commented out:

# setenv SKYBRICKS_DIR $env(DESI_ROOT)/target/skybricks/v3

SKYBRICKS_DIR is defined by desitarget.

redrock

    setenv KMP_AFFINITY disabled
    setenv MPICH_GNI_FORK_MODE FULLCOPY
    setenv OMP_NUM_THREADS 1
    setenv HDF5_USE_FILE_LOCKING FALSE

dust

Multiple packages re-define DUST_DIR, so I'm including it here, even though it's a "generic" setenv:

    # DUST_DIR
    setenv [string toupper $product]_DIR $PRODUCT_DIR
    # DUST_VERSION
    setenv [string toupper $product]_VERSION $version

fastspecfit

    setenv DUST_DIR /global/cfs/cdirs/cosmo/data/dust/v0_1
    setenv DR9_DIR /global/cfs/cdirs/desi/external/legacysurvey/dr9
    setenv FTEMPLATES_DIR $env(DESI_ROOT)/external/templates/fastspecfit
    setenv MPICH_GNI_FORK_MODE FULLCOPY
    setenv KMP_AFFINITY disabled
    setenv OMP_NUM_THREADS 1
    setenv MKL_NUM_THREADS 1

desimodules

These would be set after all other modules have been loaded:

    setenv DESI_SPECTRO_CALIB $env(DESI_ROOT)/spectro/desi_spectro_calib/0.6.0
    setenv DESI_SURVEYOPS $env(DESI_ROOT)/survey/ops/surveyops/tags/2.0

    # TODO: who sets DESI_SPECTRO_DARK?

    # may solve some OpenMP instabilities at NERSC
    setenv KMP_INIT_AT_FORK FALSE

desispec sets DESI_SPECTRO_DARK so that comment can be removed from the module file.

sbailey commented 3 months ago

Example of why this is somewhat tricky: I just merged desihub/desispec#2245 which reasonably wanted to update the desispec/main module to use desi_spectro_dark/latest instead of a named version. However, that has the consequence that we would need to temporarily undo that when we make the next desispec tag, which seems like a recipe for mistakes, e.g. this happened for a related case of desispec/0.63.x tagging desi_spectro_calib/trunk into the module files. As a consequence

source desi_environment.sh 24.4
--> DESI_SPECTRO_CALIB=...desi_spectro_calib/0.6.0
    because that was set in desimodules/24.4, overriding desispec/0.63.0 setting it to trunk

module swap desispec/0.63.1
--> DESI_SPECTRO_CALIB=...desi_spectro_calib/trunk
    because desispec overrode it back to trunk, which is probably not what we intended

I'm leaning towards saying that desimodules should own most of these environment variables, even for ones like DESI_SPECTRO_CALIB and DESI_SPECTRO_DARK which are only used by desispec.

Thoughts from others?

weaverba137 commented 3 months ago

@sbailey, I am interested in your suggestion, but by "these" do you mean environment variables that are very, very similar to DESI_SPECTRO_CALIB (point to a directory containing a version of a data set), or all of the environment variables in conflict listed above, or all "extra" environment variables generally?

weaverba137 commented 3 months ago

After discussion with @sbailey today:

  1. Wait until after Jura is completed before taking any action (~ 1 month).
  2. These types of environment variables should be moved to desimodules: a. Pointers to versioned data sets: DESI_SPECTRO_CALIB, DESI_SPECTRO_DARK, QN_MODEL_FILE, etc. b. NERSC performance: OMP_NUM_THREADS, etc. c. DUST_DIR could also be defined in desimodules; that module is not desiInstall-able so moving it to desimodules would remove the need to hand-maintain that module.
  3. Remove DESI_SURVEYOPS from desitree; though this is not as urgent.
  4. Other *_DIR variables not explicitly mentioned in desitarget will be left alone.