search

LSSTDESC / desc-help

DESC Computing Requests
BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

[NERSC] python path issue in SN environment #51

Closed reneehlozek closed 3 years ago

reneehlozek commented 3 years ago

Description A clear and concise description of what the issue is.

Choose all applicable topics by placing an 'X' between the [ ]:

To Reproduce Steps to reproduce the behavior:

  1. In order to get the Pippin code running in NSERC, I have the following lines in my bash_profile.sn file 
    source /global/cfs/cdirs/lsst/groups/SN/software/setup_sn_all.sh
export PIPPIN_DIR=/global/cfs/cdirs/lsst/groups/SN/pippin
export PIPPIN_BASE=$SNANA_LSST_ROOT/starterKits/pippin
export PATH=$PATH:${PIPPIN_DIR}
  2. When running pippin.sh I get an error about 'coloredLogs', a python module needed, so I added the following line to my bash_profile.sn:

export PYTHONPATH=$PYTHONPATH:/global/common/software/lsst/common/miniconda/sn-py/lib/python3.7/site-packages

  1. However now, when I try to run pippin.sh LSST_starterKit.yml from $PIPPIN_BASE aka /global/cfs/cdirs/lsst/groups/SN/snana/SURVEYS/LSST/ROOT/starterKits/pippin

I get an issue with a module not being loaded. Specifically

 File "/global/cfs/cdirs/snana/SNANA/util/submit_batch/submit_prog_fit.py", line 16, in <module>
    import f90nml

Others (e.g. @RickKessler) don't have the PYTHONPATH command, so I wanted to check if there's a workaround?

RickKessler commented 3 years ago

further info: all works fine for Bruno and me (kessler) ... Renee is executing the same setup commands, but it doesn't work for her.

johannct commented 3 years ago

usually that means a stale config in the .bashrc or whatever global config around.....

reneehlozek commented 3 years ago

I cleaned up the config previously (I made an issue about the coloredLogs before). After step 1. above my path is

/global/common/cori_cle7/software/python/3.7-anaconda-2019.10/bin:/global/common/cori_cle7/software/python/3.7-anaconda-2019.10/condabin:/opt/cray/pe/mpt/7.7.10/gni/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.53__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.10__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.41__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/opt/intel/compilers_and_libraries_2019.3.199/linux/bin/intel64:/global/common/cori_cle7/software/darshan/3.1.7/bin:/global/common/cori/software/altd/2.0/bin:/usr/common/software/bin:/usr/common/mss/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin

and after I execute step 2 my path is /usr/common/software/root/6.18.00-cle7-py3/bin:/global/common/sw/cray/cnl7/haswell/gsl/2.5/gcc/8.2.0/sr445ay/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.53__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.10__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.41__g36b56f4.ari/bin:/opt/gcc/9.3.0/bin:/global/common/software/lsst/common/miniconda/sn-py/bin:/global/common/cori_cle7/software/python/3.7-anaconda-2019.10/condabin:/opt/cray/pe/mpt/7.7.10/gni/bin:/global/common/cori_cle7/software/darshan/3.1.7/bin:/global/common/cori/software/altd/2.0/bin:/usr/common/software/bin:/usr/common/mss/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/global/cfs/cdirs/lsst/groups/SN/snana/SNANA/bin:/global/cfs/cdirs/lsst/groups/SN/snana/SNANA/util:/global/cfs/cdirs/lsst/groups/SN/pippin

But I'm not sure where to find the f90nml python modules

heather999 commented 3 years ago

Can you do a module list and show the output here? Preferably run it with a fresh login so we can see what your environment looks like at the beginning?

heather999 commented 3 years ago

@reneehlozek Can you also try repeating the commands at a fresh command line, rather than in your .bash_profile file? (Is it really .bash_profile.sn rather than .bash_profile?)

source /global/cfs/cdirs/lsst/groups/SN/software/setup_sn_all.sh
export PIPPIN_DIR=/global/cfs/cdirs/lsst/groups/SN/pippin
export PIPPIN_BASE=$SNANA_LSST_ROOT/starterKits/pippin
export PATH=$PATH:${PIPPIN_DIR}

Do you see the same type of failure in this case? And when Rick and Bruno are doing this - are they also using a .bash_profile or are they setting up in their .bashrc or at the command line manually?

reneehlozek commented 3 years ago

hi @heather999

Here is my module list before sourcing anything in .bash_profile.sn

 1) modules/3.2.11.4
  2) altd/2.0
  3) darshan/3.1.7
  4) intel/19.0.3.199
  5) craype-network-aries
  6) craype/2.6.2
  7) cray-libsci/19.06.1
  8) udreg/2.3.2-7.0.1.1_3.38__g8175d3d.ari
  9) ugni/6.0.14.0-7.0.1.1_7.40__ge78e5b0.ari
 10) pmi/5.0.14
 11) dmapp/7.1.1-7.0.1.1_4.54__g38cf134.ari
 12) gni-headers/5.0.12.0-7.0.1.1_6.32__g3b1768f.ari
 13) xpmem/2.2.20-7.0.1.1_4.14__g0475745.ari
 14) job/2.2.4-7.0.1.1_3.41__g36b56f4.ari
 15) dvs/2.12_2.2.164-7.0.1.1_13.1__g354a5276
 16) alps/6.6.58-7.0.1.1_6.10__g437d88db.ari
 17) rca/2.2.20-7.0.1.1_4.53__g8e3fb5b.ari
 18) atp/2.1.3
 19) PrgEnv-intel/6.0.5
 20) craype-haswell
 21) cray-mpich/7.7.10
 22) craype-hugepages2M

If I run those commands on a terminal I get the same issue:

ModuleNotFoundError: No module named 'coloredlogs'

and then when I add the 'extra' one that helped with coloredLogs

export PYTHONPATH=$PYTHONPATH:/global/common/software/lsst/common/miniconda/sn-py/lib/python3.7/site-packages

I get ModuleNotFoundError: No module named 'f90nml' again.

reneehlozek commented 3 years ago

hi @heather999 I kept digging, and found that both 'coloredLogs', and 'f90nml' are in this file /global/common/software/lsst/common/miniconda/sn-env/sn_env.txt

which isn't being sourced for me apparently?

I tried moving things around to a bash_profile.ext and making them aliases so that I can do from the command line, but the problem persists

heather999 commented 3 years ago

That /global/common/software/lsst/common/miniconda/sn-env/sn_env.txt is just a record of the packages requested be added to that environment. It should have nothing to do with setting up your environment. I'm looking back at your original comment here The problem is that there is a NERSC system python in your $PATH: /global/common/cori_cle7/software/python/3.7-anaconda-2019.10/bin and it is first, so it is the version of python searched. You override that by setting PYTHONPATH but it's just covering up the real problem.
When you log onto Cori initially, without changing anything in .bash_profile.sn, what does: which python show? If it isn't /usr/bin/python, then somehow your python is being updated in your .bashrc or somewhere,maybe look at the contents of $HOME/.condarc. I wouldn't expect a NERSC anaconda python to appear in your path, unless there is some module load python happening somewhere. If all else fails, I would try a module unload python and see what which python reports then.

reneehlozek commented 3 years ago

@heather999 - thank you for this final message! I found there was a bit of code in my bashrc from 'conda init' that I guess I was too scared to remove, but have done so now and things look like they are working.

Thanks again!!!