malloryprow
commented
1 month ago jevs_global_ens_chem_grid2obs_prep.sh - Bad Data
Ran with INITDATE=20241014 and DCOMINaeronet=/lfs/h2/emc/vpppg/noscrub/ho-chun.huang/dcom/prod/aeronet
Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr593/EVS/dev/drivers/scripts/prep/global_ens/jevs_global_ens_chem_grid2obs_prep.o158973348 COMOUT: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr593/evs/v2.0 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_global_ens_chem_grid2obs_prep.158973348.cbqs01
Ho-ChunHuang-NOAA
Note to developers: You must use this PR template!
Description of Changes
To avoid ERROR of corrupted input obs file, this code will screen input aeronet file for broken record (inconsistent columns per record) and write out record with consistent column number. The checked aeronet file will be used in the ASCII2NC conversion. Minor fix for sending email in the stats step for reduced GEFS grib2 files that can be corrupted (zero size) due to disk quota exceed. This is for emc parallel to spot something wrong in the prep step.
Developer Questions and Checklist
Is this a high priorty PR? If so, why and is there a date it needs to be merged by? NO
Do you have any planned upcoming annual leave/PTO? NO
Are there any changes needed for when the jobs are supposed to run? NO
[ ] The code changes follow NCO's EE2 Standards. YES
[ ] Developer's name is removed throughout the code and have used
${USER}where necessary throughout the code. YES[ ] References the feature branch for
HOMEevsare removed from the code. YES[ ] J-Job environment variables, COMIN and COMOUT directories, and output follow what has been defined for EVS. YES
[ ] Jobs over 15 minutes in runtime have restart capability. Jobs does not exceed 15 mins.
[ ] If applicable, changes in the
dev/drivers/scriptsordev/modulefileshave been made in the correspondingecf/scriptsandecf/defs/evs-nco.def? N/A[ ] Jobs contain the appropriate file checking and don't run METplus for any missing data. YES
[ ] Code is using METplus wrappers structure and not calling MET executables directly. YES
[ ] Log is free of any ERRORs or WARNINGs. YES
Testing Instructions
For prep step ONLY
The corrupted aeroent file existed on VDATE=20241014 and in personal archived directory.
Add export DCOMINaeronet=/lfs/h2/emc/vpppg/noscrub/ho-chun.huang/dcom/prod/aeronet
in
/dev/drivers/scripts/prep/global_ens/jevs_global_ens_chem_grid2obs_prep.sh for TESTING ONLY
One should see message "DEBUG :: Line 4317 has different columns number 137 vs standard 113"
and no ERROR and WARNING message in the log