JJB tropical subseasonal diags - Wheeler and Kiladis

[chengzhuzhang]

This PR is to create the first set of diagnostics for tropical sub-seasonal variability, wave number and frequency analysis from (wheeler Kiladis 1999). Original authors of the scripts are Jim Benedict and Brian Medeiros.

[chengzhuzhang]

@jjbenedict We are on good track to complete the wavenumber frequency diagram. Here is the testing results compare a 5 year model run with the IMERG output you processed: https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags/tropical_variability_model_obs/viewer/tropical_subseasonal/wavenumber-frequency/prect-norm_sym-imerg_daily/plot.html

Though I encountered errors when testing ORL/FLUT variable. The error is from the zwf_function script and reads as follows:

['/global/cfs/cdirs/e3sm/chengzhu/e3sm_diags_zppy_test_complete_run_output/v2.LR.historical_0101_20240130/post/atm/180x360_aave/ts/daily/5yr/FLUT_201001_201412.nc']

BEFORE unit conversion: Max/min of data: 348.7017   82.57347
2024-02-29 16:24:27,107 [ERROR]: core_parameter.py(_run_diag:269) >> Error in e3sm_diags.driver.tropical_subseasonal_driver
Traceback (most recent call last):
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/e3sm_diags/parameter/core_parameter.py", line 266, in _run_diag
    single_result = module.run_diag(self)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/e3sm_diags/driver/tropical_subseasonal_driver.py", line 184, in run_diag
    test = calculate_spectrum(
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/e3sm_diags/driver/tropical_subseasonal_driver.py", line 157, in calculate_spectrum
    spec_all = wf_analysis(var, **opt)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/e3sm_diags/driver/tropical_subseasonal_driver.py", line 59, in wf_analysis
    z2 = wf.spacetime_power(x, **kwargs)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/e3sm_diags/driver/utils/zwf_functions.py", line 468, in spacetime_power
    xdetr += xmean  # put the mean back in
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/xarray/core/_typed_ops.py", line 309, in __iadd__
    return self._inplace_binary_op(other, operator.iadd)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/xarray/core/dataarray.py", line 4645, in _inplace_binary_op
    f(self.variable, other_variable)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/xarray/core/_typed_ops.py", line 515, in __iadd__
    return self._inplace_binary_op(other, operator.iadd)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/xarray/core/variable.py", line 2717, in _inplace_binary_op
    self.values = f(self_data, other_data)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/dask/array/core.py", line 1593, in __array_ufunc__
    return elemwise(numpy_ufunc, *inputs, **kwargs)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/dask/array/core.py", line 4768, in elemwise
    out = _elemwise_normalize_out(out)
  File "/global/cfs/cdirs/e3sm/zhang40/conda_envs/edv290/lib/python3.10/site-packages/dask/array/core.py", line 4868, in _elemwise_normalize_out
    raise NotImplementedError(
NotImplementedError: The out parameter is not fully supported. Received type ndarray, expected Dask Array

I'm quite puzzled why the same codes works on PRECT but not FLUT, I'm wondering if this look familiar to you when using the zwf_function script? Thanks!

[jjbenedict]

@chengzhuzhang I'm trying to run zwf_plot.py locally, and I'm also getting errors with the OLR file. There may be something wrong with either the file itself or how python reads in the file. I'm testing a few things now...

[jjbenedict]

@chengzhuzhang The short answer: The file I downloaded from NOAA seems to be corrupt. OLR values in 1979 appear to be strange, and ncview crashes when I try to plot data from June 1979. I also discovered whole months of missing data in 1985. So instead, I subset the original file to 1986-01-01 to 2022-01-15. I then discovered that there are 4 consecutive missing days in December 1994, so I wrote a short python script that eliminates these missing values using linear temporal interpolation (this should be adequate for just 4 missing days). The resulting OLR file works fine with my local version of the WK script, and I expect it to work for you as well.

I just 'gave' you the updated NOAA OLR file.

After plotting the spectra, we might want to change the "raw" spectrum contour levels for OLR in zwf_plot, i.e.,

if(vari == "olr"):
      contour_levs_raw_spec  = (-0.4,-0.2,0.,0.2,0.4,0.6,0.8,1.0,1.2,1.4,1.6,1.8,2.2,2.4 )

(My original script only supported PRECT.)

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ Much, MUCH longer answer -- so we have a record: It seems that there's definitely something wrong with either the file itself or the way that python is trying to read in the OLR file. An 'ncdump -h' of the file appears normal:

(e3sm_unified_1.9.2_login) jjbenedi@login30:/global/cfs/cdirs/e3sm/benedict/support_e3sm_trop_diag/remapped_files/noaa-olr_cmip6_180x360_24hr> ncdump -hs olr.day.mean.psl_1deg_180x360.nc
netcdf olr.day.mean.psl_1deg_180x360 {
dimensions:
    lon = 360 ;
    lat = 180 ;
    time = UNLIMITED ; // (15721 currently)
...
float olr(time, lat, lon) ;
...
olr:_Storage = "chunked" ;
        olr:_ChunkSizes = 1, 1, 180 ;
        olr:_Shuffle = "true" ;
        olr:_DeflateLevel = 2 ;
        olr:_Endianness = "little" ;
...
:_Format = "netCDF-4" ;

I cannot read this file directly using python in the E3SM-unified environment... I get the following error:

(e3sm_unified_1.9.2_login) jjbenedi@login39:~/analysis/TEST_e3sm_diags_20240301_FLUTerror/auxiliary_tools/tropical_subseasonal_diags/zwf> python zwf_plot.py 

Opening olr from single file: /global/cfs/cdirs/e3sm/benedict/support_e3sm_trop_diag/remapped_files/noaa-olr_cmip6_180x360_24hr/olr.day.mean.psl_1deg_180x360.nc
Traceback (most recent call last):
  File "/global/u2/j/jjbenedi/analysis/TEST_e3sm_diags_20240301_FLUTerror/auxiliary_tools/tropical_subseasonal_diags/zwf/zwf_plot.py", line 763, in <module>
    data = get_data(fileList, vari)
  File "/global/u2/j/jjbenedi/analysis/TEST_e3sm_diags_20240301_FLUTerror/auxiliary_tools/tropical_subseasonal_diags/zwf/zwf_plot.py", line 559, in get_data
    x = ds[variablename].compute()
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/dataarray.py", line 1101, in compute
    return new.load(**kwargs)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/dataarray.py", line 1075, in load
    ds = self._to_temp_dataset().load(**kwargs)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/dataset.py", line 789, in load
    v.load()
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/variable.py", line 546, in load
    self._data = self._data.get_duck_array()
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/indexing.py", line 696, in get_duck_array
    self._ensure_cached()
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/indexing.py", line 690, in _ensure_cached
    self.array = as_indexable(self.array.get_duck_array())
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/indexing.py", line 664, in get_duck_array
    return self.array.get_duck_array()
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/indexing.py", line 557, in get_duck_array
    array = array.get_duck_array()
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/coding/variables.py", line 74, in get_duck_array
    return self.func(self.array.get_duck_array())
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/indexing.py", line 551, in get_duck_array
    array = self.array[self.key]
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/backends/netCDF4_.py", line 98, in __getitem__
    return indexing.explicit_indexing_adapter(
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/core/indexing.py", line 858, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.2_login/lib/python3.10/site-packages/xarray/backends/netCDF4_.py", line 111, in _getitem
    array = getitem(original_array, key)
  File "src/netCDF4/_netCDF4.pyx", line 4958, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 5916, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 2029, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

When ncview-ing the file, I get a warning and, although I can see the data maps for the first several months, ncview crashes when I get to around June 1979 -- also, the OLR fields just don't seem right for the 1979 maps that I can see... this is likely due to poor satellite quality back then. The ncview warning is:

(e3sm_unified_1.9.2_login) jjbenedi@login30:/global/cfs/cdirs/e3sm/benedict/support_e3sm_trop_diag/remapped_files/noaa-olr_cmip6_180x360_24hr> ncview olr.day.mean.psl_1deg_180x360.nc
...
Note: the coordinates attribute for variable olr is being ignored,
since it specifies a variable (time) that has 1 effective dims (an effective dim has a size greater than 1)
I am not set up to handle cases with coordinate mapping using anything other than 0 or 2 effective dims
Note: no Ncview app-defaults file found, using internal defaults
calculating min and maxes for olr...
netcdf_fi_get_data: error on nc_get_vara_float call
cdfid=65536   variable=olr
start, count:
[0]: 176  1
[1]: 0  180
[2]: 0  360
NetCDF: HDF error

From the top of this post, you can see olr:_DeflateLevel = 2 ;. I've had many problems previously when using NCO with compressed files, so I tried to uncompress the file but got an error:

(e3sm_unified_1.9.2_login) jjbenedi@login30:/global/cfs/cdirs/e3sm/benedict/support_e3sm_trop_diag/remapped_files/noaa-olr_cmip6_180x360_24hr> ncks -O -L 0 olr.day.mean.psl_1deg_180x360.nc olr.day.mean.psl_1deg_180x360_decompressed.nc
ERROR: nco_get_vara() failed to nc_get_vara() variable "olr"
ERROR NC_EHDFERR Error at HDF5 layer
HINT: NC_EHDFERR errors indicate that the HDF5-backend to netCDF is unable to perform the requested task. NCO can receive this devilishly inscrutable error for a variety of possible reasons including: 1) The run-time dynamic linker attempts to resolve calls from the netCDF library to the HDF library with an HDF5 libhdf5.a that is incompatible with the version used to build NCO and netCDF. 2) The file system does not allow the HDF5 flock() function, as of HDF5 1.10.x, to enable multiple processes to open the same file for reading, a feature known as SWMR (Single Write Multiple Read). The fix is to disable the HDF5 flock() by setting an environment variable thusly: "export HDF5_USE_FILE_LOCKING=FALSE". 3) An incorrect netCDF4 library implementation of a procedure (e.g., nc_rename_var()) in terms of HDF function calls (e.g., HDF5Lmove()) manifests an error or inconsistent state within the HDF5 layer. This often occurs during renaming operations (https://github.com/Unidata/netcdf-c/issues/597). 4) Attempting to compress or decompress a netCDF4 dataset with a non-standard (i.e., non-DEFLATE) filter when the requisite shared library to encode/decode that compression filter is not present in either the default location (/usr/local/hdf5/lib/plugin) or in the user-configurable location referred to by the HDF5_PLUGIN_PATH environment variable. One can determine if missing plugin libraries are the culprit by dumping the hidden attributes of the dataset with, e.g., ncks --hdn -m in.nc or ncdump -s -h in.nc. Any variables with (hidden) "_Filter" attributes require the corresponding shared libraries to be located in HDF5_PLUGIN_PATH. Some HDF5 implementations (at least MacOSX with MacPorts as of 20200907) may also require explicitly setting the plugin path in the environment, even for the default location! To test this, re-try your NCO command after doing this: "export HDF5_PLUGIN_PATH=/usr/local/hdf5/lib/plugin". 5) Bad vibes.
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_get_vara()
nco_err_exit(): ERROR Error code is -101. Translation into English with nc_strerror(-101) is "NetCDF: HDF error"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

I was wondering if the problem could somehow be related to the OLR file size (3.3GB), so I used ncks to select only a ~5-yr segment (around years 2011-2015). The ncks command was successful, and python reads in the data correctly... so the data compression does not appear to be the problem. Given that ncview crashed around June 1979 and that the 1979 OLR values look odd, I used ncks to only select 1980-01-01 to 2022-01-15 and make a new OLR file. Python reads that file in perfectly. However, I discovered several months of missing data in 1985, and a few missing days in 1994. I suggest we use the period 1986-01-01 to 2022-01-15 (or less) for an analysis window, and I will linearly interpolate those few missing days in 1994. The ZWF script cannot be passed missing data due to the FFTs.

[chengzhuzhang]

@jjbenedict It turned out that when calculating spectral power for ORL from model, when calculating time averaging, the program somehow didn't load data in memory, by adding .load() as here solved the issue NotImplementedError: The out parameter is not fully supported. Received type ndarray, expected Dask Array, I found earlier. It is interesting that this error only happens for ORL from model, but not PRECT from model. I suspect that the ORL/FLUT array somehow is larger than PRECT and triggered using Dask array. I have to check with @tomvothecoder on this..

On the other hand, thanks for the new dataset! I was able to run through all 3 datasets you have provided and generated reasonable results:https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags/tropical_variability_model_obs/viewer/tropical_subseasonal/index.html. The last thing I need to fix is the diff ratio plot for raw spectrum are not coming up correctly, I will continue trouble shooting..

[tomvothecoder]

the program somehow didn't load data in memory, by adding .load() as here solved the issue NotImplementedError: The out parameter is not fully supported. Received type ndarray, expected Dask Array

.load() expects a Dask Array to convert to np.ndarray. In your case, the underlying data is already an np.ndarray so .load() raises that error.

I suspect that the ORL/FLUT array somehow is larger than PRECT and triggered using Dask array. I have to check with @tomvothecoder on this..

If the dataset is opened with xr.open_dataset(), then the underlying data will be np.ndarray unless you explicitly pass a chunks arg. On the other hand, xr.open_mfdataset() will load the data as Dask Array until .load()/.compute() is called, which then converts them to np.ndarray.

[jjbenedict]

@chengzhuzhang Thanks, glad to hear you've discovered the error source. One thing I noticed: the figure html links on https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags/tropical_variability_model_obs/viewer/tropical_subseasonal/index.html are broken for PRECT and OLR (they work for U850). I can see plots when I mouse over the PRECT and OLR "Plot" link, but when I click on the link its says that it cannot find the page.

[chengzhuzhang]

@tomvothecoder thanks for the explanation! I checked the code again, the dataset is opended with xr.open_mfdataset(), in the PRECT case, there was a step to convert units, where data was loaded and converted to np.ndarray before hand. But .load() is then needed for other variables FLUT, U850 to convert them from dask array.

[chengzhuzhang]

@chengzhuzhang Thanks, glad to hear you've discovered the error source. One thing I noticed: the figure html links on https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags/tropical_variability_model_obs/viewer/tropical_subseasonal/index.html are broken for PRECT and OLR (they work for U850). I can see plots when I mouse over the PRECT and OLR "Plot" link, but when I click on the link its says that it cannot find the page.

Thank you for reviewing the plots. One more thing to fix...

[chengzhuzhang]

I think I have fixed the diff ratio plot, but still can't figure out the issue with the viewer. Only last variable "FLUT" (in this example) has html page generated with plots and correctly linked to viewer: https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags_try2/tropical_variability_model_obs/viewer/tropical_subseasonal/index.html The figures for PRECT are also created and can be pre-viewed, but the html pages for plots are not generated.

@tomvothecoder Could you help to review the viewer code? I may need fresh eyes to stare at the codes at this point. If it helps, the run directory can be found here: /global/cfs/cdirs/e3sm/www/chengzhu/tests/tropical_diags_try2/tropical_variability_model_obs/.

[tomvothecoder]

@tomvothecoder Could you help to review the viewer code? I may need fresh eyes to stare at the codes at this point. If it helps, the run directory can be found here: /global/cfs/cdirs/e3sm/www/chengzhu/tests/tropical_diags_try2/tropical_variability_model_obs/.

@chengzhuzhang Can you re-base this branch on the latest main to reduce the file diffs? Thanks

[chengzhuzhang]

@jjbenedict hey Jim, this PR is ready for review. I made a run for v2 historical (2001-2010) to compare with some of your earlier results (https://acme-climate.atlassian.net/wiki/spaces/EWCG/pages/3500343313/Manuscript+outline+and+planning+space). The e3sm diags run can be found here: https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags_try2/tropical_variability_model_obs/viewer/tropical_subseasonal/wavenumber-frequency-prect/prect-norm_sym-imerg_daily/plot.html I feel that the obs data from the e3sm_diags run matches with obs from your case better. While the model run is a little bit off. Could you help check if the results are still reasonable?

[chengzhuzhang]

@tomvothecoder The code is also ready for review. I ran pre-commit locally and fixed all the problems, but the CI/CD is still failing.. Could you help take a look? Thank you.

[jjbenedict]

@chengzhuzhang The WK plot differences between your new E3SM Diags run and those earlier plots you compared to are not alarming, but the amount of noise in the v2 E3SM Diags plot is a little unexpected. Just to note here:

• The "earlier" WK observational plots were based on TRMM precipitation (we're using IMERG, which will give a slightly different result)
• The "earlier" v2 plots first interpolated the precip data to a 2.5° grid (I assume E3SM Diags interpolates to a 1° grid?). I don't think this would strongly alter the WK spectra, though, since we're focused on zonal wavenumbers -15 to +15... relatively "large-scale"

Do you happen to have the file(s) containing PRECT interpolated to the 1° grid that E3SM Diags uses to make the model WK plots? If so, I could point my original zwf_plot.py script at that/those files and remake the WK plots for comparison.

[chengzhuzhang]

@jjbenedict thank you for analyzing the results and helping me understand. Yes, the E3SM Daigs used all 1 degree data. Here are the remapped model data: /global/cfs/cdirs/e3sm/chengzhu/e3sm_diags_zppy_test_complete_run_output/v2.LR.historical_0101_20240130/post/atm/180x360_aave/ts/daily/5yr . I created them from zppy and they can be directly used by E3SM Diags, though, you may have to adjust the original script to read them..thanks for helping testing!

[tomvothecoder]

I think I have fixed the diff ratio plot, but still can't figure out the issue with the viewer. Only last variable "FLUT" (in this example) has html page generated with plots and correctly linked to viewer: portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags_try2/tropical_variability_model_obs/viewer/tropical_subseasonal/index.html The figures for PRECT are also created and can be pre-viewed, but the html pages for plots are not generated.

@tomvothecoder Could you help to review the viewer code? I may need fresh eyes to stare at the codes at this point. If it helps, the run directory can be found here: /global/cfs/cdirs/e3sm/www/chengzhu/tests/tropical_diags_try2/tropical_variability_model_obs/.

I think the viewer issue might be fixed with this PR review suggestion.

[chengzhuzhang]

@chengzhuzhang The WK plot differences between your new E3SM Diags run and those earlier plots you compared to are not alarming, but the amount of noise in the v2 E3SM Diags plot is a little unexpected. Just to note here:

* The "earlier" WK observational plots were based on TRMM precipitation (we're using IMERG, which will give a slightly different result)

* The "earlier" v2 plots first interpolated the precip data to a 2.5° grid (I assume E3SM Diags interpolates to a 1° grid?). I don't think this would strongly alter the WK spectra, though, since we're focused on zonal wavenumbers -15 to +15... relatively "large-scale"

Do you happen to have the file(s) containing PRECT interpolated to the 1° grid that E3SM Diags uses to make the model WK plots? If so, I could point my original zwf_plot.py script at that/those files and remake the WK plots for comparison.

@jjbenedict it turned out to be a bug in subsetting time in my code, and only one year of data was used instead of 10 years in the original run for E3SM v2. With accurate subsetting of (2001-2010) data, I get less noiser plot and is more resemble those you posted on confluence page. I updated the run here: https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags_try2/tropical_variability_model_obs/viewer/tropical_subseasonal/wavenumber-frequency-prect/prect-norm_sym-imerg_daily/plot.html

[jjbenedict]

@chengzhuzhang Yes, that latest plot looks much better. Can you point me to a revised list of output plots (like https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags_try2/tropical_variability_model_obs/viewer/tropical_subseasonal/wavenumber-frequency-prect/, but now with the bug fixes)? Thanks!

[chengzhuzhang]

@jjbenedict thanks for checking the plots. This url links to the full list of plots with the bug fix..

[jjbenedict]

Hi @chengzhuzhang I see (1) a plotting bug, and (2) I think we might need to add a few lines of code to extend the dispersion curves.

Regarding the plotting bug: it looks like all of the plot files produced, both "symmetric" and "asymmetric", use the symmetric shallow water dispersion curves. Can you adjust so that the asymmetric plots use the asymmetric dispersion curves?

Regarding the dispersion curve addition: This is my error. For a few of the wave types, the dispersion curves become undefined at zonal wavenumber = 0. I have a quick fix for this, in zwf_plot.py. Is this something I can do by pulling the most recent tag, modifying it, and then pushing it back? I could do this today, but a little guidance would make things go faster... I don't have much experience (or a vague memory) of the git pull-push process.

[chengzhuzhang]

@jjbenedict Thanks for catching the errors! I can place a fix quickly for both, if you could send me the code change for the zwf_plot.py...so that we don't step on each other...

Otherwise, I believe you can try following steps to update the zwf_plot.py script:

git clone https://github.com/E3SM-Project/e3sm_diags.git
cd e3sm_diags
git fetch origin jjb_tropical_subseasonal_diags
git checkout jjb_tropical_subseasonal_diags
 vi auxiliary_tools/tropical_subseasonal_diags/zwf/zwf_plot.py
(make change to the script)
git commit -am "update original zwf_plot.py" --no-verify
git push origin jjb_tropical_subseasonal_diags 

In this case, I will review the change, add the fixes and test. But we can also re-visit and brush off on git skills and other development steps when including the lag-coefficient set...

[chengzhuzhang]

@tomvothecoder while testing this feature, I notice that the newly included file: zwf_function.py was not correctly installed. Could you help to figure out how to properly include this file? Thanks!

[tomvothecoder]

@tomvothecoder while testing this feature, I notice that the newly included file: zwf_function.py was not correctly installed. Could you help to figure out how to properly include this file? Thanks!

I ran the following commands and was able to import that module locally. Maybe check to make sure you're running make install to delete the old build and install the new build.

mamba env create -f conda/dev.yml -n e3sm_diags_dev_732
mamba activate e3sm_diags_dev_732
make install
python

Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from e3sm_diags.driver.utils import zwf_functions
>>> zwf_functions
<module 'e3sm_diags.driver.utils.zwf_functions' from '/gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_diags/e3sm_diags/driver/utils/zwf_functions.py'>
>>> 

[chengzhuzhang]

@tomvothecoder thanks for troubleshooting! Now I see, there was a bad import, the problem is fixed with https://github.com/E3SM-Project/e3sm_diags/pull/732/commits/0cc310c832529478fe45bfa74ea04077530c5423

[chengzhuzhang]

Hi @jjbenedict I have updated the READ.me for including the wavenumber frequency plot. https://github.com/E3SM-Project/e3sm_diags/pull/732/commits/a8bcf9e9c5e9740d3569fe9032d57b19dccae33a Could you let me know if other contributors I should add?

I have fixed the dispersion curves for the asymmetric plot, I can help to update the plotting for the other issue if you haven't started. After that, I think this branch is ready to merge. (I also fixed most review comments @tomvothecoder provided. )

[chengzhuzhang]

@jjbenedict I'm implementing this new set with zppy, and ran some tests with the v3 picontrol data. And here is the result I have: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.zhang40/E3SMv3_trop/v3.LR.piControl/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_0001-0050/viewer/tropical_subseasonal/index.html It looks like v3 is doing a nice job in simulating the power spectra...

I think we are very close to merge this feature. Let me know if anything I can help to refine the plots..

[jjbenedict]

@chengzhuzhang Apologies for the delay, I'm working on the dispersion curve fix now (having dispersion curve lines cross the zero zonal wavenumber...

[jjbenedict]

@chengzhuzhang I cloned the repo, made edits to zwf_plot.py, and committed the changes (locally). I tried to push back to origin but got an authentication error... I think I need to set up the authentication in github, and I know I've done this before and can probably figure it out. I'll try to do this over the weekend, otherwise early Monday.

I had one or two other questions (I'll save those for Monday because I have to leave the office now), but one other minor note: For the asymmetric plots you shared with me, none show the "MRG" wave label. I can see the line in zwf_plot.py that adds the "MRG" label to the plot, but for some reason it didn't show up. The "MRG" label -does- show up in plots that I make using my local version of zwf_plot.py... and the code appears to match what I see from the E3SM Diags repo. We can revisit this Monday.

[chengzhuzhang]

@jjbenedict thanks for the updates. I will be out of office next week, it looks like we are near completion of this feature. I will get back and we can work towards releasing the week of April 8th.

I re-organized the plotting into e3sm_diags/plot/cartopy/tropical_subseasonal_plot.py it could be something I messed up in displaying the labels...

[jjbenedict]

@chengzhuzhang @tomvothecoder I noticed that my Github personal access token had expired again, so I renewed it. I also noticed that I am no longer in the e3sm_diags UNIX group, but I was still able to push my commits to the remote repo.

Where we're at now: The commits I just made fix minor bugs related to the shallow water dispersion curves on the plots (some "jumped" the zero zonal wavenumber value). These edits also plot the dispersion lines that were missing on some plots, and label the lines. I've tested these code mods on my local version of the script and they work. @chengzhuzhang could you test these in your workflow? Note that I decided to NOT plot dispersion curves for the "background" spectral plots, following the WK99 paper... but we could decide to change this down the road.

Next:

1. @chengzhuzhang Please check on why the "MRG" line label is missing on all asymmetric plots.
2. Plot change: Can we label all colorbar contours? Or, alternatively, have colorbar contour labels be symmetric about 0?
3. Plot change: Max end of PRECT "raw" contours: OLD: [..., 0.0, 0.1, 0.2] NEW: [..., 0.0, 0.2, 0.4]
4. Plot change: Change U850 "normalized" contours to: 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.7, 1.9, 2.3, 2.7, 3.3, 3.9
5. Plot change: Change U850 "raw" contours to: -1.6, -1.4, -1.2, -1.0, -0.8, -0.6, -0.5, -0.4, -0.3, -0.2, -0.1, 0.0, 0.2, 0.4, 0.6

The suggested contour changes should reduce "off-scaling" of large plotted values. And, re: (4) and (5), I hadn't fully tested the WK plotting routines with U850... hopefully these contour changes improve the plots. I think we're extremely close to a final product!!

[chengzhuzhang]

Hey @jjbenedict I believe all the issues and suggestions from your comment (https://github.com/E3SM-Project/e3sm_diags/pull/732#issuecomment-2030368391) are addressed in my recent commits. Please have a review of the new plots.

Please review the results here: https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags_subsetting/tropical_variability_model_obs_refine/viewer/tropical_subseasonal/index.html

[chengzhuzhang]

@tomvothecoder I'm puzzled about he CI/CD failure, it seems to be downloading data for integration testing was problematic. There is a 502 Bad Getway error. Not sure if it is a server problem, re-ran won't resolve. The file in question is in /lcrc/group/e3sm/public_html/e3sm_diags_test_data/integration/expected/integration_test_images/prov, any suggestion? or maybe you could try re-set the permission of the files?

[tomvothecoder]

@tomvothecoder I'm puzzled about he CI/CD failure, it seems to be downloading data for integration testing was problematic. There is a 502 Bad Getway error. Not sure if it is a server problem, re-ran won't resolve. The file in question is in /lcrc/group/e3sm/public_html/e3sm_diags_test_data/integration/expected/integration_test_images/prov, any suggestion? or maybe you could try re-set the permission of the files?

I experimented different combinations of permissions (including 777) for the prov directory and they all still raise a 502 Bad Gateway. I also copied prov as prov_cp and that directory also raises the same error. I'm not 100% what has changed since the last time the build was successful. Maybe re-running the integration tests and replacing prov might help.

https://web.lcrc.anl.gov/public/e3sm/e3sm_diags_test_data/integration/expected/prov_cp/

[chengzhuzhang]

@jjbenedict I'm preparing a release candidate to test with e3sm unified new release. I will merge this PR for now, but let me know if you'd like any change, we can update in the final release. Thank you!

[tomvothecoder]

Awesome work @chengzhuzhang!

[jjbenedict]

@chengzhuzhang and @tomvothecoder -- Super exciting and thanks so much for your help with making this a reality!! The plots look awesome, Jill. And thanks for being patient with my pickiness and novice github knowledge. I expect implementation of the next round of tropical subseasonal variability plots (lag correlation, possibly others) to go more smoothly and quickly. Let me know if anything else comes up regarding the WK plots.

[chengzhuzhang]

@jjbenedict Thank you so much for your contribution. And I think this set is in great shape with you careful review and thoughtful requests! We are looking forward to working with you again for new enhancements for the tropical variability set.