Upgrade to spack-stack 1.8.0 for running standalone

[malloryprow]

EMC_verif-global standalone will update to using the spack-stack modules in the next spack-stack release (June-July timeframe).

[DavidHuber-NOAA]

FYI I tried giving this a shot for release 1.7.0, but cartopy was causing issues with the gsi-addon Python version. When the unified-env spack-stack environment upgrades to Python 3.11.x, we should be able to get the needed Python modules into the gsi-addon spack-stack environment. The target is the following release (1.8.0) in June-July. This work is being tracked in https://github.com/JCSDA/spack-stack/issues/942.

[malloryprow]

Great, thank you! It will be great to use the spack-stack for running EMC_verif-global standalone! I'll update the title to reflect 1.8.0.

[InnocentSouopgui-NOAA]

I successfully ran using spack-stack 1.6.0 on S4. This requires two things:

1. loading python packages (matplolib, panda, numpy, cartopy, etc. ) as modules;
2. update the python code to change some features that were remove from matplolib in version 3.4
   • matplotlib.axes.SubplotBase.colNum -> ax.get_subplotspec().colspan.start
   • matplotlib.axes.SubplotBase.rowNum -> ax.get_subplotspec().rowspan.start
   • matplotlib.axes.SubplotBase.is_last_row() -> ax.get_subplotspec().is_last_row()
   • matplotlib.axes.SubplotBase.is_first_col() -> ax.get_subplotspec().is_first_col()

See the changes on my fork, branch migration/spack/stack https://github.com/InnocentSouopgui-NOAA/EMC_verif-global/tree/migration/spack-stack

The change I made in the code is not backward compatible, however, it is easy to adjust it for backward compatibility with matplotlib 3.3 and below.

There is still a couple of things to fix

• I ran a test where I get 751 plot files instead of 770 with the change to spack stack
• some numbers (tick labels, etc.) are printed on way more digits, see the images below (one with spack sstack and the other with hpc-stack). Possibly something changed from using single precision to double precision. Those "ill"-formatted numbers appear on a handfull of plots here and there. I think it should be a fairly easy fix.

Plotted with spack stack (see the number on top of the top left panel)

Plotted with hpc-stack

We can also notice a big change in the data as presented: Look inside the South America for the bottom left panel of each of the plot; the difference is big. For the two bottom panel, there seems to be no negative value with HPC stack, while there are negative values with spack-stack. Those are plotted from the same data, using the same version of MET and METplus, the major difference is in the external libraries. However, this one is an outlier, I did not look at all the images, but this is the only one that I saw with such a difference.

Another plot with ill-formatted numbers

[malloryprow]

This is interesting, thanks for putting this out @InnocentSouopgui-NOAA. What if you do ncview on the two files? I guess I first want to make sure all is well with the MET and METplus installations. If it is the external libraries, I would suspect using ncview the files would look the same. If you need help investigating and there is a way to get this all on Hera, I can help investigate.

[InnocentSouopgui-NOAA]

@malloryprow, I put the tes data on hera at /scratch2/NESDIS/nesdis-rdo1/Innocent.Souopgui/verif_global_input_data I also did a quick check with neview and did not spot any obvious difference. I will take the time to rerun both tests and see.

[malloryprow]

Here is what I got running on Hera. It is using hpc-stack.

[InnocentSouopgui-NOAA]

@malloryprow, that looks consistent with what I got using spack stack on S4.

[InnocentSouopgui-NOAA]

I did a little more investigations, I think that the version of the plotting library (matplotlib) has something to do with the resulting plot. The plot with everything from spack stack is the correct one. See the screenshot below. I wrote a simple python script to compute the difference T_ten - ctrl and save it back to netcdf. This is after verif_global run. The left is from the spack stack run and the right is from hpc-stack run. They look identical, with the negatives that we are seeing in the plot with external libraries from spack stack.

[InnocentSouopgui-NOAA]

We might need to move quickly to spack stack.

[malloryprow]

Tank you for the investigation @InnocentSouopgui-NOAA!

@DavidHuber-NOAA Is there a status update for spack-stack v1.8.0?

[DavidHuber-NOAA]

@malloryprow The tentative release date is the end of July/beginning of August. This is the issue that is tracking the Python upgrade: https://github.com/JCSDA/spack-stack/issues/1038.

[malloryprow]

@InnocentSouopgui-NOAA Which version of spack stack were you using the produced the good plots? This was on S4 correct?

[InnocentSouopgui-NOAA]

@InnocentSouopgui-NOAA Which version of spack stack were you using the produced the good plots? This was on S4 correct?

I used spack-stack 1.6.0 on S4. You can see the module file https://github.com/InnocentSouopgui-NOAA/EMC_verif-global/blob/migration/spack-stack/modulefiles/emc_verif_global_s4.lua

[malloryprow]

Thanks! That is interesting. I was never able to get the standalone working with 1.6.0 on Hera. I had to us the gsi-addon because that is where MET and METplus were installed but then that version's python didn't have the needed python modules like cartopy to use.

[InnocentSouopgui-NOAA]

I also built MET using spack-stack to be able to do that on S4. So I have MET built with spack-stack on s4 as well.

[InnocentSouopgui-NOAA]

Also important to say that with spack-stack, python packages come as modules. So you have to load cartopy, matplotlib. numpy, etc. like other modules

module load py-netcdf4
module load py-numpy
module load py-pandas
module load py-matplotlib
module load py-cartopy

before it is available for use in python.

[DavidHuber-NOAA]

I have a test installation of PR https://github.com/JCSDA/spack-stack/pull/1252 on Hera. I will be trying it out today to run step2 plotting (g2g, g2o, and precip) as well as maps2d.

If you would like to play around with it, I believe this is the load sequence that will be needed:

module use /scratch1/NCEPDEV/global/David.Huber/SPACK/ss_python_3117/envs/met913_down/install/modulefiles/Core
module load stack-intel
module load stack-python
module load py-netcdf4
module load py-numpy
module load py-pandas
module load py-matplotlib
module load py-cartopy
module load wgrib2
module load met/9.1.3
module load metplus/3.1.1
module load prod-util
module load grib-util

[malloryprow]

Ah, that is great news, thanks!

[DavidHuber-NOAA]

@malloryprow I have been able to run verif-global through the step1 jobs successfully, but the step2 jobs and maps2d are failing. Would you mind taking a look at my branch and seeing if I have something misconfigured? For instance, the log file /scratch1/NCEPDEV/global/David.Huber/para/stmp/verif_standalone/tmp/verif_global.2444030/grid2grid_step2/metplus_output/logs/master_metplus_grid2grid_step2_anom_plotbyVALID_SAL1L2_TM P_G002_runon20240820180947.log shows an error when attempting to calculate the axis. Perhaps I have bad data to work with?

My working directory is available here on Hera: /scratch1/NCEPDEV/global/David.Huber/VERIF/verif-global_ss180 and I created the following branch on GitHub that includes the configs: https://github.com/DavidHuber-NOAA/EMC_verif-global/tree/feature/ss_180.

[malloryprow]

I'll try to give it a look here today. I have some EVS PRs I need to work through that are my priority right now. I'll definitely look at it this week.

[DavidHuber-NOAA]

Sounds good, thanks @malloryprow!

[malloryprow]

I ran a test for grid2grid step 2 and I'm seeing the same errors your seeing. I'm using different data though (operational GFS stat files).

[malloryprow]

Are we sure everything is okay with this python installation? I can even get a simple matplotlib plot to be created with it.

https://matplotlib.org/stable/gallery/lines_bars_and_markers/simple_plot.html