search

NOAA-EMC / EMC_verif-global

Global Forecast System (GFS) verification package using MET and METplus
8 stars 14 forks source link

Create Modulefile for S4 #115

Closed DavidHuber-NOAA closed 3 months ago

DavidHuber-NOAA commented 1 year ago

A modulefile should be created for S4 and module loading moved out of load_modules.sh.

DavidHuber-NOAA commented 1 year ago

@malloryprow I'm working on this now, so feel free to assign me.

DavidHuber-NOAA commented 1 year ago

The step1 jobs are all running fine with the new module file, but when running step2 jobs in standalone mode, the plotting is going extremely slow. For instance, the grid2grid plotting, when run for a single experiment on 2022040200, is timing out. The plotting logs are advancing, but again very slowly. I'm not sure if it is related, but I am seeing several messages about a very large number of tick marks. For instance, the log file verif_global.405174/grid2grid_step2/metplus_output/logs/master_metplus_grid2grid_step2_anom_plotbyVALID_SAL1L2_HGT_NHX_runon20230411141952.log has several messages including

Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
04/11 14:29:04.403 (plot_time_series.py:282) INFO: Working on forecast lead 240000 and forecast variable HGT P250
04/11 14:29:04.404 (plot_time_series.py:356) INFO: Reading in model data
04/11 14:29:04.419 (plot_time_series.py:482) INFO: Calculating and plotting statistics
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
04/11 14:30:58.355 (plot_time_series.py:863) INFO: Saving image as /scratch/users/dhuber/verif_global/tmp/verif_global.405174/grid2grid_step2/metplus_output/plot_by_VALID/make_plots/SAL1L2_HGT_NHX/grid2grid/anom/images/acc_valid00Z_HGT_P250_fhr24_G002NHX.png

The resulting plot is below image

I'm not sure if this is slowing the plotting down. @malloryprow Do you know if this is an issue on other machines? I'm happy to send more logs/plots over to Hera/Orion if you want to take a look.

malloryprow commented 1 year ago

@DavidHuber-NOAA Usually when the plotting jobs are running extremely so it is because of bad stats data (originating from the model analysis or model forecasts). If there is an very large difference between the model analysis and model forecast, this will cause potentially wide range of values for the stats, and the code is designed to iteratively adjust the y-axis to avoid the lines from being in the legend (so it is legible and clean). I've experienced this a few times before.

However, here I think the code might be struggling with the fact you are just plotting a single day. The code isn't really set up to just plot a single point since it really isn't useful and doesn't make sense statistically.

DavidHuber-NOAA commented 1 year ago

@malloryprow OK, that makes sense. What would you consider to be the minimum period of time that makes sense?

malloryprow commented 1 year ago

3 days should be good

InnocentSouopgui-NOAA commented 3 months ago

@DavidHuber-NOAA, Any update on this issue? I can take it over if you have a lot of other things. I have S4's users who are having trouble with the stand alone verif-global.

DavidHuber-NOAA commented 3 months ago

@InnocentSouopgui-NOAA Please feel free to take over. I haven't been able to work on this for some time.