feature/EVSv2-RTOFS_additions2

SamiraArdani-NOAA commented 2 months ago

Note to developers: You must use this PR template!

Description of Changes

This PR addresses one of the fixes and addition items for EVSv2-RTOFS regarding the variable name changes through the entire steps of this EVS component. Here is a list of changes:

1- $RUN=ocean for each step of EVSv2-RTOFS was defined to be consistent with other EVS components. 2- $RUN was defined in all j-jobs. 3- $RUNsmall was renamed to $RUN in stats j-job and all stats scripts; and 4- For all observation types, variable $OBTYPE was used instead of $RUN throughout all scripts.

Please include a summary of the changes and the related GitHub issue(s). Please also include relevant motivation and context.

There is no GitHub issue for this change. The motivation was to update the EVSv2-RTOFS to be consistent with other EVS components in terms of variable names.

Developer Questions and Checklist

Is this a high priority PR? If so, why and is there a date it needs to be merged by? No.
Do you have any planned upcoming annual leave/PTO? No.
Are there any changes needed for when the jobs are supposed to run? I would like to run all steps with PDYm3.
[x] The code changes follow NCO's EE2 Standards.
[x] Developer's name is removed throughout the code and have used ${USER} where necessary throughout the code.
[x] References the feature branch for HOMEevs are removed from the code.
[x] J-Job environment variables, COMIN and COMOUT directories, and output follow what has been defined for EVS.
[x] Jobs over 15 minutes in runtime have restart capability.
[x] If applicable, changes in the dev/drivers/scripts or dev/modulefiles have been made in the corresponding ecf/scripts and ecf/defs/evs-nco.def?
[x] Jobs contain the appropriate file checking and don't run METplus for any missing data.
[ ] Code is using METplus wrappers structure and not calling MET executables directly.
[x] Log is free of any ERRORs or WARNINGs.

Testing Instructions

Please include testing instructions for the PR assignee. Include all relevant input datasets needed to run the tests.

To test this PR, you need to run all driver scripts for all three steps: prep, stats and plots. 1- Clone my fork in your local and checkout branch feature/EVSv2-RTOFS_additions2 2- ln -s fix directory. 3- Replace $USER with emc.vpppg in driver scripts. 4- Run prep step: $EVS/dev/drivers/scripts/prep/rtofs/jevs_rtofs_prep.sh 5- Run all stats scripts: $EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_argo_grid2obs_stats.sh $EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_aviso_grid2grid_stats.sh $EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_ghrsst_grid2grid_stats.sh $EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_ndbc_grid2obs_stats.sh $EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_osisaf_grid2grid_stats.sh $EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_smap_grid2grid_stats.sh $EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_smos_grid2grid_stats.sh 6- Run all plots scripts: $EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_argo_grid2obs_last60days_plots.sh $EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_aviso_grid2grid_last60days_plots.sh $EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_ghrsst_grid2grid_last60days_plots.sh $EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_ndbc_grid2obs_last60days_plots.sh $EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_osisaf_grid2grid_last60days_plots.sh $EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_smap_grid2grid_last60days_plots.sh $EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_smos_grid2grid_last60days_plots.sh $EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_headline_grid2grid_last90days_plots.sh

SamiraArdani-NOAA commented 2 months ago

@AliciaBentley-NOAA, Thanks for reviewing the PR. I needed that lowercase obtype because OBTYPE is required for .config plots whereas the obtype is used for .png files naming. So, that was the reason I defined lowercase obtype in addition to the uppercase OBTYPE.

malloryprow commented 1 month ago

@SamiraArdani-NOAA Can the associated RUN changes done in the drivers scripts be done in the ecf scripts too?

SamiraArdani-NOAA commented 1 month ago

@SamiraArdani-NOAA Can the associated RUN changes done in the drivers scripts be done in the ecf scripts too?

Done!

malloryprow commented 1 month ago

Disregard my last post about the rtofs prep being done! I messed something up!

malloryprow commented 1 month ago

Can you do a sync fork when you get the chance?

malloryprow commented 1 month ago

I'll pick this PR back up again on Monday! I took the rainy afternoon to get some trainings out of the way.

malloryprow commented 1 month ago

Morning! Running prep!

jevs_rtofs_prep.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/prep/rtofs/jevs_rtofs_prep.o156757373 COMOUT: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/evs/v2.0 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_prep.156757373.cbqs01

SamiraArdani-NOAA commented 1 month ago

@malloryprow, No ERROR or WARNING was found in .o file. The output results was compared with emc.vpppg (evs_beta5) and they all look as expected.

malloryprow commented 1 month ago

Awesome! On to stats. Is it okay to point COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/evs_beta5/v2.0? What VDATE should I use to test?

SamiraArdani-NOAA commented 1 month ago

@malloryprow, Yes, you can point to emc.vpppg/evs_beta5/v2.0. Please use VDATE=20240927.

malloryprow commented 1 month ago

Stats jobs are submitted! All submitted with VDATE=20240927. COMOUT for all is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/evs/v2.0.

jevs_rtofs_argo_grid2obs_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_argo_grid2obs_stats.o156774529 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_argo_grid2obs_stats.156774529.cbqs01

jevs_rtofs_aviso_grid2grid_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_aviso_grid2grid_stats.o156774552 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_aviso_grid2grid_stats.156774552.cbqs01

jevs_rtofs_ghrsst_grid2grid_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_ghrsst_grid2grid_stats.o156774651 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_ghrsst_grid2grid_stats.156774651.cbqs01

jevs_rtofs_ndbc_grid2obs_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_ndbc_grid2obs_stats.o156774776 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_ndbc_grid2obs_stats.156774776.cbqs01

jevs_rtofs_osisaf_grid2grid_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_osisaf_grid2grid_stats.o156774922 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_osisaf_grid2grid_stats.156774922.cbqs01

jevs_rtofs_smap_grid2grid_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_smap_grid2grid_stats.o156775053 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_smap_grid2grid_stats.156775053.cbqs01

jevs_rtofs_smos_grid2grid_stats.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/stats/rtofs/jevs_rtofs_smos_grid2grid_stats.o156775122 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_smos_grid2grid_stats.156775122.cbqs01

SamiraArdani-NOAA commented 1 month ago

@malloryprow, $COMOUT for OBTYPEs smos, smap, ghrsst, osisaf, argo and aviso look identical to evs_beta5 emc.vpppg parallel. No ERROR or WARNING was observed in the .o files. We just need to wait for ndbc runs to be finished.

SamiraArdani-NOAA commented 1 month ago

@malloryprow, It looks like ndbc stats runs were finished. I Checked the outputs. They are identical to those in evs_beta5 emc.vpppg parallel.

malloryprow commented 1 month ago

Plots time! I'll point to the beta5 for COMIN. Should I do VDATE=20240928?

SamiraArdani-NOAA commented 1 month ago

@malloryprow, Thanks! Yes, you can use VDATE=20240928.

malloryprow commented 1 month ago

plots ran with VDATE=20240928. COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/evs/v2.0.

jevs_rtofs_argo_grid2obs_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_argo_grid2obs_last60days_plots.o156854312 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_argo_grid2obs_last60days_plots.156854312.cbqs01

jevs_rtofs_aviso_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_aviso_grid2grid_last60days_plots.o156854314 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_aviso_grid2grid_last60days_plots.156854314.cbqs01

jevs_rtofs_ghrsst_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_ghrsst_grid2grid_last60days_plots.o156854315 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_ghrsst_grid2grid_last60days_plots.156854315.cbqs01

jevs_rtofs_ndbc_grid2obs_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_ndbc_grid2obs_last60days_plots.o156854328 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_ndbc_grid2obs_last60days_plots.156854328.cbqs01

jevs_rtofs_osisaf_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_osisaf_grid2grid_last60days_plots.o156854341 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_osisaf_grid2grid_last60days_plots.156854341.cbqs01

jevs_rtofs_smap_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_smap_grid2grid_last60days_plots.o156854345 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_smap_grid2grid_last60days_plots.156854345.cbqs01

jevs_rtofs_smos_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_smos_grid2grid_last60days_plots.o156854346 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_smos_grid2grid_last60days_plots.156854346.cbqs01

jevs_rtofs_headline_grid2grid_last90days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_headline_grid2grid_last90days_plots.o156854317 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_headline_grid2grid_last90days_plots.156854317.cbqs01

SamiraArdani-NOAA commented 1 month ago

@malloryprow, No ERROR was found in .o files. WARNING was only observed due to missing smos data for 20240928 in smos log file. The size of tar balls are smaller than those in emc.vpppg. I assume it is because the COMIN points to evs_beta5 stats and the number of stats runs in evs_beta5 is smaller than in evs. Do you run parallel for evs_beta5 plots as well?

SamiraArdani-NOAA commented 1 month ago

@malloryprow, Let me investigate more by untarring the tarballs.

malloryprow commented 1 month ago

No plots yet for the beta5 parallel. You are correct though, they would be smaller since there is only about about 10 days of stats so far.

SamiraArdani-NOAA commented 1 month ago

@malloryprow, would you please run the osisaf plots for VDATE=20240928 by pointing to evs parallel stats?

malloryprow commented 1 month ago

jevs_rtofs_osisaf_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_osisaf_grid2grid_last60days_plots.o156872882 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_osisaf_grid2grid_last60days_plots.156872882.cbqs01 COMOUT: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/evs/v2.0

SamiraArdani-NOAA commented 1 month ago

@malloryprow, the number of .png files is identical to emc.vpppg. Would you please check if you point $COMIN to evs (not evs_beta5)? I wonder why the size of tar file is different?

malloryprow commented 1 month ago

Got COMIN right this time.

jevs_rtofs_osisaf_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_osisaf_grid2grid_last60days_plots.o156881479 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_osisaf_grid2grid_last60days_plots.156881479.cbqs01 COMOUT: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/evs/v2.0

SamiraArdani-NOAA commented 1 month ago

@malloryprow, The outputs look fine. Now the osisaf tarball size is identical to that in emc.vpppg. Thanks for running it by pointing to evs parallel $COMIN. If you do not mind, would it be possible to run the rest of the plots by pointing to evs parallel (instead of evs_beta5), VDATE=20240928? I just want to make sure all runs are comparable with emc.vpppg? Thanks again!

malloryprow commented 1 month ago

plots ran with VDATE=20240928 with COMIN to beta4 parallel. COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/evs/v2.0.

jevs_rtofs_argo_grid2obs_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_argo_grid2obs_last60days_plots.o156944401 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_argo_grid2obs_last60days_plots.156944401.cbqs01

jevs_rtofs_aviso_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_aviso_grid2grid_last60days_plots.o156944410 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_aviso_grid2grid_last60days_plots.156944410.cbqs01

jevs_rtofs_ghrsst_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_ghrsst_grid2grid_last60days_plots.o156854315 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_ghrsst_grid2grid_last60days_plots.156854315.cbqs01

jevs_rtofs_ndbc_grid2obs_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_ndbc_grid2obs_last60days_plots.o156944523 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_ndbc_grid2obs_last60days_plots.156944523.cbqs01

jevs_rtofs_osisaf_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_osisaf_grid2grid_last60days_plots.o156944530 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_osisaf_grid2grid_last60days_plots.156944530.cbqs01

jevs_rtofs_smap_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_smap_grid2grid_last60days_plots.o156944553 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_smap_grid2grid_last60days_plots.156944553.cbqs01

jevs_rtofs_smos_grid2grid_last60days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_smos_grid2grid_last60days_plots.o156944561 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_smos_grid2grid_last60days_plots.156944561.cbqs01

jevs_rtofs_headline_grid2grid_last90days_plots.sh

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr568/EVS/dev/drivers/scripts/plots/rtofs/jevs_rtofs_headline_grid2grid_last90days_plots.o156944518 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_rtofs_headline_grid2grid_last90days_plots.156944518.cbqs01

SamiraArdani-NOAA commented 1 month ago

@malloryprow, Plots look good. Thanks for doing this test. The little difference in tatball size now make sense because the plot timing has one day shift compare to emc.vpppg for VDATE=20240928.

malloryprow commented 1 month ago

Thanks Samira!

NOAA-EMC / EVS