search

E3SM-Project / e3sm-unified

A metapackage for a unified anaconda environment for analyzing results from the Energy Exascale Earth System Model (E3SM).
BSD 3-Clause "New" or "Revised" License
8 stars 8 forks source link

e3sm-unified 1.2.3 issue on NERSC #27

Closed vanroekel closed 5 years ago

vanroekel commented 5 years ago

I tried using e3sm-unified for MPAS-Analysis on NERSC this morning and am getting an error, see below

lvroekel:MPAS-Analysis$ module load e3sm-unified/1.2.3
lvroekel:MPAS-Analysis$ ./run_mpas_analysis --list
Traceback (most recent call last):
  File "./run_mpas_analysis", line 37, in <module>
    from mpas_analysis.shared.plot.plotting import _register_custom_colormaps, \
  File "/global/u2/l/lvroekel/MPAS-Analysis/mpas_analysis/shared/plot/plotting.py", line 30, in <module>
    from mpl_toolkits.basemap import Basemap
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/base/envs/e3sm_unified_1.2.3_py2.7_nox/lib/python2.7/site-packages/mpl_toolkits/basemap/__init__.py", line 155, in <module>
    pyproj_datadir = os.environ['PROJ_LIB']
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/base/envs/e3sm_unified_1.2.3_py2.7_nox/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'PROJ_LIB'

e3sm-unified/1.2.2 works fine. If this issue is more appropriate for the MPAS-analysis repo, let me know.

xylar commented 5 years ago

@vanroekel, thanks for pointing this out. I can reproduce the issue. If you access the anaconda environment the way that they prefer (see below) this doesn't happen so I guess the issue is that loading the environment as a module doesn't set required environment variables. The following worked for me:

source  /global/project/projectdirs/acme/software/anaconda_envs/${NERSC_HOST}/base/etc/profile.d/conda.sh
conda activate e3sm_unified_1.2.3_py2.7_nox

I'll look into whether there's a feasible way to set the required environment variables as part of the module, but I suspect this may be a problem with more packages than just basemap.

xylar commented 5 years ago

@vanroekel, I'm trying to figure out how to simplify things a bit. Could you try the following (in a fresh shell on Edison) when you get a chance?

source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_unified.sh
run_mpas_analysis --list

This should tell us whether the little script I made can load the environment correctly. Hopefully sourcing that script rather than loading a module won't end up being too onerous. I don't know what else to do because modules seem simply to be an infeasible option at this point.

vanroekel commented 5 years ago

I tried, but it looks like I don't have permissions to run the script.

xylar commented 5 years ago

Sorry about that. Please try again when you have a chance.

vanroekel commented 5 years ago

It works for me. Thanks for the quick fix! I will say I have no problem with the source activate method. This seems like a very reasonable way to load python, at least to me.

xylar commented 5 years ago

Yep, this just does that under the hood so users who don't want to think about it can just source a script and that script will know what the latest version is.

I'll make these helper scripts on all the machines tomorrow and then update the instructions and send out a follow-up email explaining moving away from modules (again) and apologizing for the moving target.

xylar commented 5 years ago

Addressed by requesting that users activate the environment directly with the load_latest_e3sm_unified* script.