python updates generate incompatible ctsm_pylib environment

[wwieder]

Brief summary of bug

run_neon won't work with newly created ctsm_pylib environments.

General bug information

CTSM version you are using: [output of git describe] master, ctsm5.2.007

Does this bug cause significantly incorrect results in the model's science? [Yes / No] no, but it renders run_neon useless for new users

Configurations affected: [Fill this in if known.] run_neon, but possible other python tools that depend on ctsm_pylib

Details of bug

Out of the box run_neon fails with the following error. This was using a ctsm_pylib environment @katierocci created yesterday. The case ran fine when we used an older version of ctsm_pylib that I had instead.

We also were able to run with for another user who created the python environment a few weeks ago, using the same CTSM tag. This me think it's a recent update to derecho that's causing the incompatibility?

Important details of your setup / configuration so we can reproduce the bug

./run_neon --neon-sites BART --output-root /glade/derecho/scratch/$USER/neon_cases

Traceback (most recent call last):

  File "./run_neon", line 45, in <module>

    from ctsm.site_and_regional.run_neon import main

  File "/glade/work/krocci/CTSM/tools/site_and_regional/../../python/ctsm/site_and_regional/run_neon.py", line 66, in <module>

    from ctsm.download_utils import download_file

  File "/glade/work/krocci/CTSM/tools/site_and_regional/../../python/ctsm/download_utils.py", line 5, in <module>

    import requests

  File "/glade/work/krocci/conda-envs/ctsm_pylib/lib/python3.7/site-packages/requests/__init__.py", line 164, in <module>

    from .api import delete, get, head, options, patch, post, put, request

  File "/glade/work/krocci/conda-envs/ctsm_pylib/lib/python3.7/site-packages/requests/api.py", line 11, in <module>

    from . import sessions

  File "/glade/work/krocci/conda-envs/ctsm_pylib/lib/python3.7/site-packages/requests/sessions.py", line 15, in <module>

    from .adapters import HTTPAdapter

  File "/glade/work/krocci/conda-envs/ctsm_pylib/lib/python3.7/site-packages/requests/adapters.py", line 60, in <module>

    from urllib3.contrib.socks import SOCKSProxyManager

  File "/glade/work/krocci/conda-envs/ctsm_pylib/lib/python3.7/site-packages/urllib3/contrib/socks.py", line 75, in <module>

    class _TYPE_SOCKS_OPTIONS(typing.TypedDict):

AttributeError: module 'typing' has no attribute 'TypedDict'

(ctsm_pylib) krocci@derecho3:/glade/work/krocci/CTSM/tools/site_and_regional

[wwieder]

for anyone needing a functional ctsm_pylib environment on NCAR systems, you can use this YML file until we have a fix. /glade/u/home/wwieder/ctsm_pylib_ww.yml

[wwieder]

@samsrabin or @slevis-lmwg is this something you can solve easily?

[samsrabin]

Hmm. the error in import requests sounds like what @ekluzek and @glemieux are experiencing on Izumi, but it looks like this is happening for Katie on Derecho...

Generally nothing having to do with Python environments is easy, but some more diagnostic info might be helpful. @katierocci, could you point us to the directory from which you set up the Python environment? And do you know the exact command you used?

[katierocci]

@samsrabin I set up the python environment in /glade/work/krocci/CTSM and ran the command ./py_env_create

[wwieder]

You can see the diffs between my older ctsm_pylib environment that Katie was able to run with and the one that's created currently on derecho by diff-ing the yml files in my home directory

diff  /glade/u/home/wwieder/ctsm_pylib_June24.yml /glade/u/home/wwieder/ctsm_pylib_ww.yml

Note, Katie's current env was created from ctsm_pylib_ww.yml. My current env is reflected in the *June24.yml file

[wwieder]

@samsrabin wondered if we could move beyond python 3.7, but this isn't really possible on izzumi. With Izzumi going away maybe we have an opportunity to migrate to a newer python version.

[wwieder]

@samsrabin will try pinning all packages in the conda_env_ctsm_py.txt file and get rid of the conda_env_ctsm_py_latest.txt file.

[samsrabin]

I was unable to reproduce this, so I asked Katie to run py_env_create again with the created environment's name changed to ctsm_pylib_test. That ended up working, so I guess something weird must have gone wrong in her original attempt. I'll close this issue.

However, in my testing, I did run into #2563. I'll take on that PR (#2607) because I'm personally annoyed by it now!