If an ob has an "incoming" QC value that denotes "do not use",
then filter assigns the DART QC = 6 and assigns MISSING_R8 to the data values.
Users might expect that the ob will appear in Matlab diagnostic pictures, with its associated QC,
in the same way that an ob which fails the forward operator appears in the pictures with QC=4 associated with it.
But the QC=6 ob is not displayed.
This happens because obs_seq_to_netcdf uses the MISSING_R8 to assign NetCDF's missing_value
to QC=6 obs in epoch.### (aka obs_seq_final.nc) This is where control switches from QC to data value.
Read_obs_netcdf.m uses the NetCDF missing_values to assign Matlab NaNs,
which are not displayed in the link_obs pictures.
Error Message
None. Users may or may not notice that some obs are not displayed.
Which model(s) are you working with?
Any
Version of DART
v10.8.2, but this behavior extends back to at least 2019.
Have you modified the DART code?
No
Build information
This will happen on any machine.
Discussion
I believe that the general opinion is that users should be able to choose whether QC=6 obs should be displayed.
In real obs assimilations there may be so many that they interfere with seeing the assimilated obs.
In OSSEs (and real obs) it may be valuable to see the them, at least some of the time.
The minimal fix appears to be adding a namelist variable to obs_seq_to_netcdf.nml
to control whether those obs are displayed and making the QC be the controlling variable
instead of the data value. But it would be more convenient for users to put the control
in the Matlab scripts, so that obs_seq_to_netcdf doesn't need to be rerun to change the display mode.
When displaying them, having a data value of MISSING_R8 would be awkward (color scales, etc.),
so we may want to preserve the original value if there is one, or assign a value that can be plotted
in a way that identifies it as QC=6. That might be the right context for using MISSING_R8 in the data.
This might require a separate call to scatter on the same plot with just the QC=6 points.
There are probably other considerations that I'm missing.
Describe the bug
If an ob has an "incoming" QC value that denotes "do not use", then filter assigns the DART QC = 6 and assigns MISSING_R8 to the data values. Users might expect that the ob will appear in Matlab diagnostic pictures, with its associated QC, in the same way that an ob which fails the forward operator appears in the pictures with QC=4 associated with it. But the QC=6 ob is not displayed.
This happens because obs_seq_to_netcdf uses the MISSING_R8 to assign NetCDF's missing_value to QC=6 obs in epoch.### (aka obs_seq_final.nc) This is where control switches from QC to data value. Read_obs_netcdf.m uses the NetCDF missing_values to assign Matlab NaNs, which are not displayed in the link_obs pictures.
Error Message
None. Users may or may not notice that some obs are not displayed.
Which model(s) are you working with?
Any
Version of DART
v10.8.2, but this behavior extends back to at least 2019.
Have you modified the DART code?
No
Build information
This will happen on any machine.
Discussion
I believe that the general opinion is that users should be able to choose whether QC=6 obs should be displayed. In real obs assimilations there may be so many that they interfere with seeing the assimilated obs. In OSSEs (and real obs) it may be valuable to see the them, at least some of the time. The minimal fix appears to be adding a namelist variable to obs_seq_to_netcdf.nml to control whether those obs are displayed and making the QC be the controlling variable instead of the data value. But it would be more convenient for users to put the control in the Matlab scripts, so that obs_seq_to_netcdf doesn't need to be rerun to change the display mode.
When displaying them, having a data value of MISSING_R8 would be awkward (color scales, etc.), so we may want to preserve the original value if there is one, or assign a value that can be plotted in a way that identifies it as QC=6. That might be the right context for using MISSING_R8 in the data. This might require a separate call to scatter on the same plot with just the QC=6 points.
There are probably other considerations that I'm missing.