Open wlin7 opened 9 months ago
Does it mean the plotting codes are currently intended only for time series from the beginning of a simulation? Is it easy to make it work for any selected sub-period?
No, that works. The cfg I use for testing starts at 1850 (https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/test_complete_run_chrysalis.cfg). But I wasn't sure if that was the simulation start date. So, I ran a cfg starting at 1860 instead and it generated plots ok: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_test_debug_www/546-change-start-yr/v2.LR.historical_0201/global_time_series/global_time_series_1860-1870_results/v2_historical_0201_glb.pdf
cfg:
[default]
case = v2.LR.historical_0201
constraint = ""
dry_run = "False"
environment_commands = ""
input = "/lcrc/group/e3sm/ac.forsyth2//E3SMv2/v2.LR.historical_0201"
input_subdir = archive/atm/hist
mapping_file = "map_ne30pg2_to_cmip6_180x360_aave.20200201.nc"
# To run this test, edit `output` and `www` in this file, along with `actual_images_dir` in test_complete_run.py
output = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_debug_output/546-change-start-yr/v2.LR.historical_0201"
partition = "debug"
qos = "regular"
www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_test_debug_www/546-change-start-yr"
[ts]
active = True
e3sm_to_cmip_environment_commands = "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_to_cmip_20240229"
walltime = "00:30:00"
years = "1860:1864:2",
[[ atm_monthly_glb ]]
# Note global average won't work for 3D variables.
frequency = "monthly"
input_files = "eam.h0"
input_subdir = "archive/atm/hist"
mapping_file = "glb"
years = "1860:1870:5",
[[ land_monthly_glb ]]
frequency = "monthly"
input_files = "eam.h0"
input_subdir = "archive/atm/hist"
mapping_file = "glb"
[mpas_analysis]
active = True
anomalyRefYear = 1860
climo_years ="1860-1864", "1865-1870",
enso_years = "1860-1864", "1865-1870",
mesh = "EC30to60E2r2"
parallelTaskCount = 6
partition = "compute"
qos = "regular"
ts_years = "1860-1864", "1860-1870",
walltime = "00:30:00"
[global_time_series]
active = True
climo_years ="1860-1864", "1865-1870",
experiment_name = "v2.LR.historical_0201"
figstr = "v2_historical_0201"
moc_file=mocTimeSeries_1860-1870.nc
ts_num_years = 5
ts_years = "1860-1864", "1860-1870",
walltime = "00:30:00"
years = "1860-1870",
I'm having a hard time figuring out what's causing this error. TypeError: 'NoneType' object is not subscriptable
is not particularly helpful.
Also worth noting is that the error is coming from cdutil
which needs to be removed/replaced anyway as part of the CDAT migration (#346/#519)
So something is happening in this block, specifically at exp["annual"]["ohc"] = ts.globalAnnual("ohc")
:
https://github.com/E3SM-Project/zppy/blob/4db3e24c7082a26f7c915690abd1d89bbd52c1c4/zppy/templates/coupled_global.py#L585-L590
But I'm not sure why that's failing.
$ cd /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts
$ grep -v "OK" *status
global_time_series_0051-1400.status:WAITING 465551
global_time_series_0101-1400.status:ERROR (5)
global_time_series_0101-1450.status:WAITING 465658
global_time_series_0901-1100.status:ERROR (5)
global_time_series_0901-1150.status:ERROR (3)
global_time_series_1001-1150.status:ERROR (3)
global_time_series_1001-1200.status:ERROR (5)
global_time_series_1001-1300.status:ERROR (5)
global_time_series_1001-1350.status:ERROR (5)
global_time_series_1001-1400.status:ERROR (5)
global_time_series_1001-1500.status:ERROR (5)
global_time_series_1001-1550.status:ERROR (5)
global_time_series_1001-1700.status:ERROR (5)
global_time_series_1001-1900.status:ERROR (5)
global_time_series_1001-1950.status:ERROR (5)
global_time_series_1001-2000.status:ERROR (5)
global_time_series_1501-1800.status:ERROR (5)
global_time_series_1501-1900.status:ERROR (5)
global_time_series_1501-1950.status:ERROR (3)
global_time_series_1501-2000.status:ERROR (5)
mpas_analysis_ts_0001-1400_climo_1351-1400.status:ERROR (1)
mpas_analysis_ts_0001-1600_climo_1551-1600.status:ERROR (1)
mpas_analysis_ts_0051-1400_climo_1351-1400.status:ERROR (1)
mpas_analysis_ts_0101-1450_climo_1401-1450.status:ERROR (1)
$ grep "OK" global_time_series_*.status
global_time_series_0001-0300.status:OK
global_time_series_0001-0400.status:OK
global_time_series_0001-0450.status:OK
global_time_series_0001-0500.status:OK
global_time_series_0001-0600.status:OK
global_time_series_0001-0650.status:OK
global_time_series_0001-0700.status:OK
global_time_series_0001-0750.status:OK
global_time_series_0001-0800.status:OK
global_time_series_0001-0850.status:OK
global_time_series_0001-0900.status:OK
global_time_series_0001-0950.status:OK
global_time_series_0001-1000.status:OK
global_time_series_0001-1050.status:OK
global_time_series_0001-1100.status:OK
global_time_series_0001-1150.status:OK
global_time_series_0001-1200.status:OK
global_time_series_0001-1250.status:OK
global_time_series_0001-1300.status:OK
global_time_series_0001-1350.status:OK
It does look like only the global_time_series
jobs beginning at 0001
ran successfully.
There are two types of errors:
ERROR (5)
is what you describe here.
Traceback (most recent call last):
File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 651, in <module>
run_by_region(sys.argv)
File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 647, in run_by_region
run(parameters, rgn)
File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 588, in run
exp["annual"]["ohc"] = ts.globalAnnual("ohc")
File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/readTS.py", line 65, in globalAnnual
v = cdutil.YEAR(v)
File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 1416, in get
File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 477, in get
File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 751, in monthBasedSlicer
TypeError: 'NoneType' object is not subscriptable
Corresponds to https://github.com/E3SM-Project/zppy/blob/4db3e24c7082a26f7c915690abd1d89bbd52c1c4/zppy/templates/global_time_series.bash#L88-L95
$ grep "ERROR (5)" global_time_series_*.status
global_time_series_0101-1400.status:ERROR (5)
global_time_series_0901-1100.status:ERROR (5)
global_time_series_1001-1200.status:ERROR (5)
global_time_series_1001-1300.status:ERROR (5)
global_time_series_1001-1350.status:ERROR (5)
global_time_series_1001-1400.status:ERROR (5)
global_time_series_1001-1500.status:ERROR (5)
global_time_series_1001-1550.status:ERROR (5)
global_time_series_1001-1700.status:ERROR (5)
global_time_series_1001-1900.status:ERROR (5)
global_time_series_1001-1950.status:ERROR (5)
global_time_series_1001-2000.status:ERROR (5)
global_time_series_1501-1800.status:ERROR (5)
global_time_series_1501-1900.status:ERROR (5)
global_time_series_1501-2000.status:ERROR (5)
ERROR (3)
Traceback (most recent call last):
File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/bin/cdscan", line 1843, in <modu\
le>
main(sys.argv)
File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/bin/cdscan", line 1309, in main
vartime[0],
File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdm\
s2/axis.py", line 2212, in __getitem__
raise IndexError('index out of bounds')
IndexError: index out of bounds
Corresponds to https://github.com/E3SM-Project/zppy/blob/4db3e24c7082a26f7c915690abd1d89bbd52c1c4/zppy/templates/global_time_series.bash#L67-L75
$ grep "ERROR (3)" global_time_series_*.status
global_time_series_0901-1150.status:ERROR (3)
global_time_series_1001-1150.status:ERROR (3)
global_time_series_1501-1950.status:ERROR (3)
What happened?
The following error occur when trying to generate global time series plot that does not start from year 1 of a simulation (v3.LR.piControl-spinup). This happens after e3sm_diags and mpas-analysis for the covered period have been completed properly. Same errors also occur for other subperiods, such as global_time_series_0901-1100, global_time_series_1001-1400, regardless of the length of the time series. Does it mean the plotting codes are currently intended only for time series from the beginning of a simulation? Is it easy to make it work for any selected sub-period?
Examples can be found under /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts
What machine were you running on?
chrysalis
Environment
v2.3.1rc1
What command did you run?
Copy your cfg file
What jobs are failing?
What stack trace are you encountering?