search

NOAA-EMC / RTOFS_GLO

0 stars 1 forks source link

Bad AMSR data causes incomplete assimilation #43

Open DanIredell-NOAA opened 7 months ago

DanIredell-NOAA commented 7 months ago

A zero byte AMSR file (only on the dev machine) resulted in no AMSR data being ingested as the AMSR assimilation failed upon processing this file. No warning was issued either. (the file was OK on the prod machine so the production run ran OK).

So.. maybe need to do a sanity check (ncdump -k) on ALL netCDF files. Currently only sst files are checked ($DCOMROOT/YYYYMMDD/sst).

DanIredell-NOAA commented 7 months ago

Current files checks in v2.3.4 $DCOMROOT/YYYYMMDD/sst/*.nc

Current files checked in branch bugfix: $DCOMROOT/YYYYMMDD/wtxtbul/satSSS/SMAP/*.h5 $DCOMROOT/YYYYMMDD/wtxtbul/satSSS/SMOS/*.nc

Files not checked: $DCOMROOT/YYYYMMDD/seaice/pda/*.nc $DCOMROOT/YYYYMMDD/wgrdbul/adt/*.nc $DCOMROOT/YYYYMMDD/wgrdbul/ndbc/*.nc

Scripts that will need to be modified to check these: seaice/pda - ush/rtofs_ncoda_ice_qc.sh wgrdbul/ndbc - ush/rtofs_ncoda_vel_qc.sh

Unfortunately for wgrdbul/adt, the data-path is exported to the NCODA program as SSH_DATA_DIR/YYYYMMDD/SSH_DATA_DIR2 (where SSH_DATA_DIR2 is wgrdbul/adt) and the files to read are . It would be preferable if the NCODA program read the filenames from a designated file (like it does for other data files), and then it can be pre-checked for sanity before-hand (in ush/rtofs_ncoda_ssh_qc.sh).

DanIredell-NOAA commented 7 months ago

Modified ush/rtofs_ncoda_ice_qc.sh, ush/rtofs_ncoda_vel_qc.sh, and ush/rtofs_ncoda_ssh_qc.sh to call ncdump on all input netCDF files. If ncdump fails then those files will not be read by the NCODA routines. Also a warning message will be output indicating that the file was unreadable.

In ush/rtofs_ncoda_ssh_qc.sh, instead of removing the invalid file from a list of files, good files are copied to a temp directory that NCODA will read from (SSH_DATA_DIR).