Closed gthompsnJCSDA closed 4 months ago
I agree with the proposed code change.
I'd like to highlight a practical aspect concerning the direct assimilation of negative dBZ values using variational techniques. We've observed that including a significant number of negative dBZ values tends to slow down the convergence of the cost function significantly. Adjusting from 10 dBZ to -20 dBZ, compared to adjusting from 30 dBZ to 0 dBZ, results in a relatively minor change in the mixing ratio. Yet, it occupies a similar portion of the cost function adjustment effort. To ensure faster convergence and efficient iteration, we typically treat negative dBZ (except for missing value) values as zero dBZ in our preprocessing procedures in practice. This approach may compromise accuracy slightly, but it significantly enhances efficiency.
@chengsiliu The timing of your review comment was impeccable. I was formulating an Email discussion to you when this comment arrived. Let's continue the discussion in Email. Obviously, we can permit whatever values we want to pass through a converter and deal with them differently in QC. That is ultimately a user decision in my opinion, and we should not be short-cutting the data because of past work if the future provides a means to discover something new and the code can operate the same regardless.
@gthompsnJCSDA, please, adjust your reference file if you are happy with the changes in the converter. The unit test is failing with:
ECCODES ERROR : Truncating time: non-zero seconds(41) ignored
DIFFER : LENGTHS OF RECORDS : Location (5419491) <> Location (249186)
DIFFER : LENGTHS : DIMENSION : Location : 5419491 <> 249186
DIFFER : VARIABLE : Location : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : height : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : latitude : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : longitude : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : dateTime : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : height : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : latitude : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : longitude : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
@gthompsnJCSDA, please, adjust your reference file if you are happy with the changes in the converter. The unit test is failing with:
ECCODES ERROR : Truncating time: non-zero seconds(41) ignored DIFFER : LENGTHS OF RECORDS : Location (5419491) <> Location (249186) DIFFER : LENGTHS : DIMENSION : Location : 5419491 <> 249186 DIFFER : VARIABLE : Location : SIZE : 5419491 <> 249186 DIFFER : VARIABLE : height : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999 DIFFER : VARIABLE : latitude : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999 DIFFER : VARIABLE : longitude : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999 DIFFER : VARIABLE : dateTime : SIZE : 5419491 <> 249186 DIFFER : VARIABLE : height : SIZE : 5419491 <> 249186 DIFFER : VARIABLE : latitude : SIZE : 5419491 <> 249186 DIFFER : VARIABLE : longitude : SIZE : 5419491 <> 249186 DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186 DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186 DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
@gthompsnJCSDA, this comment is valid still.
@gthompsnJCSDA reference file still needs update looks like
@gthompsnJCSDA reference file still needs update looks like
Fixed the testoutput
file. Awaiting CI-tests to ensure all pass.
@gthompsnJCSDA CI is not working properly do you think this one is ready otherwise
@gthompsnJCSDA CI is not working properly do you think this one is ready otherwise
Someone needs to explain to me why the JEDI unit test fails, because I don't understand why.
@gthompsnJCSDA CI is not working properly do you think this one is ready otherwise
Someone needs to explain to me why the JEDI unit test fails, because I don't understand why.
The test did not run. There was an error setting up the CI to test the code. It's fixed now.
Description
The MRMS converter was chopping off data with reflectivity less than -25 dBZ. Instead, we can capture the special values of -99 as valid data areas at below detectable signal limits. That leaves only the other missing value of -999 in areas that are not covered by the radar composite - either terrain blocked or geometry of radar scans. This Bugfix now allows users to include or exclude based on the lower limit of reflectivity (variable called
min_dbz
can be set using the--skip
command line option during data conversion). By default it is set to -35 dBZ and the special value of -99 is changed to be -34.5 for keeping those points for the output file. In practice, it appears there is never echo in the MRMS product below -30 dBZ so the special value keeps data in a tighter range than -99.Issue(s) addressed
Resolves #1474
Dependencies
Should be no dependencies
Impact
Should be no impact to other repos.
Checklist