CIME paths need to be updated in template scripts

[billsacks]

The template scripts (template.case.run, etc.) have paths to CIME tools that are no longer correct. In testing on izumi with a recent version of cime, this gave the following error at the start of the test job (immediately after the submitted job began to run):

Traceback (most recent call last):
  File "/var/spool/torque/mom_priv/jobs/559137.izumi.cgd.ucar.edu.SC", line 21, in <module>
    from standard_script_setup import *
ModuleNotFoundError: No module named 'standard_script_setup'

I can't figure out why I'm getting this problem on izumi but not on cheyenne, and @ekluzek reports not seeing it on either machine. One reason that you potentially wouldn't get this error is if you are using an older model checkout where cime/scripts/Tools used to exist at one point before you updated cime: in this case, you can still have a stale .pyc file in cime/scripts/Tools that can allow this to work. I encountered this issue with a fresh checkout. (But @ekluzek says he tried it with a fresh checkout.) On cheyenne, it looks like the sys.path already contains the needed entry (cime/CIME/Tools) on entry to the .case.test script, so the addition of a wrong path (cime/scripts/Tools) does no harm; but I can't tell why cime/CIME/Tools is already in sys.path on entry to .case.test on cheyenne but not on izumi. (I see there are some places in CIME that modify PYTHONPATH, and maybe this modification has different behavior in different situations????)

Fix incoming.

[ekluzek]

Although I inexplicably can't reproduce the problem, when I do the python code created from the template file for .case.test for a case and do it by hand, I do replicate the above result.

It also looked like to me that if I send case.submit from an existing test case, I should see the behavior, but that doesn't happen for me either. Below is the top of the .case.test for my example case on cheyenne. I think what happens is that sys.path is inherited from case.submit and sent into the batch system.

I do see the bad behavior if I submit the .case.test by hand, since it doesn't get a correct sys.path on entry to the batch script.

#!/usr/bin/env python3
#PBS -N test.SMS_D_Lm1_Mmpi-serial.CLM_USRDAT.I1PtClm50SpRs.cheyenne_intel.clm-USUMB_nuopc.20220519_144203_fpjew7
#PBS  -r n 
#PBS  -j oe 
#PBS  -V 
#PBS  -S /bin/bash  
#PBS  -l select=1:mpiprocs=1:ompthreads=1
"""
This is the system test submit script for CIME. This should only ever be called
by case.submit when on batch system. This script only exists as a way of providing
batch directives. Use case.submit from the command line to run your case.

DO NOT RUN THIS SCRIPT MANUALLY
"""
import os, sys
os.chdir( '/glade/scratch/erik/ctsm5.1.dev096/cime/scripts/SMS_D_Lm1_Mmpi-serial.CLM_USRDAT.I1PtClm50SpRs.cheyenne_intel.clm-USUMB_nuopc.20220519_144203_fpjew7')

_LIBDIR = os.path.join("/glade/scratch/erik/ctsm5.1.dev096/cime", "scripts", "Tools")
sys.path.append(_LIBDIR)

from standard_script_setup import *

[ekluzek]

@billsacks created a PR for this, and I verified that with the new tag ccs_config_cesm0.0.36 I can do the python code in the .case.test file and have it work correctly. That was the only way I was able to replicate this problem, and #46 fixes it. This has been a weird situation, but I'm glad it's working now.