Closed DavidHuber-NOAA closed 3 months ago
@malloryprow I'm working on this now, so feel free to assign me.
The step1 jobs are all running fine with the new module file, but when running step2 jobs in standalone mode, the plotting is going extremely slow. For instance, the grid2grid plotting, when run for a single experiment on 2022040200, is timing out. The plotting logs are advancing, but again very slowly. I'm not sure if it is related, but I am seeing several messages about a very large number of tick marks. For instance, the log file verif_global.405174/grid2grid_step2/metplus_output/logs/master_metplus_grid2grid_step2_anom_plotbyVALID_SAL1L2_HGT_NHX_runon20230411141952.log
has several messages including
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
04/11 14:29:04.403 (plot_time_series.py:282) INFO: Working on forecast lead 240000 and forecast variable HGT P250
04/11 14:29:04.404 (plot_time_series.py:356) INFO: Reading in model data
04/11 14:29:04.419 (plot_time_series.py:482) INFO: Calculating and plotting statistics
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
Locator attempting to generate 73825 ticks ([701335.0, ..., 775159.0]), which exceeds Locator.MAXTICKS (1000).
04/11 14:30:58.355 (plot_time_series.py:863) INFO: Saving image as /scratch/users/dhuber/verif_global/tmp/verif_global.405174/grid2grid_step2/metplus_output/plot_by_VALID/make_plots/SAL1L2_HGT_NHX/grid2grid/anom/images/acc_valid00Z_HGT_P250_fhr24_G002NHX.png
The resulting plot is below
I'm not sure if this is slowing the plotting down. @malloryprow Do you know if this is an issue on other machines? I'm happy to send more logs/plots over to Hera/Orion if you want to take a look.
@DavidHuber-NOAA Usually when the plotting jobs are running extremely so it is because of bad stats data (originating from the model analysis or model forecasts). If there is an very large difference between the model analysis and model forecast, this will cause potentially wide range of values for the stats, and the code is designed to iteratively adjust the y-axis to avoid the lines from being in the legend (so it is legible and clean). I've experienced this a few times before.
However, here I think the code might be struggling with the fact you are just plotting a single day. The code isn't really set up to just plot a single point since it really isn't useful and doesn't make sense statistically.
@malloryprow OK, that makes sense. What would you consider to be the minimum period of time that makes sense?
3 days should be good
@DavidHuber-NOAA, Any update on this issue? I can take it over if you have a lot of other things. I have S4's users who are having trouble with the stand alone verif-global.
@InnocentSouopgui-NOAA Please feel free to take over. I haven't been able to work on this for some time.
A modulefile should be created for S4 and module loading moved out of load_modules.sh.