JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA
8 stars 2 forks source link

[Bugfix] Allow treatment of radar echo below detectable limit but present in data as compared to regions of no mrms data coverage #1475

Closed gthompsnJCSDA closed 4 months ago

gthompsnJCSDA commented 4 months ago

Description

The MRMS converter was chopping off data with reflectivity less than -25 dBZ. Instead, we can capture the special values of -99 as valid data areas at below detectable signal limits. That leaves only the other missing value of -999 in areas that are not covered by the radar composite - either terrain blocked or geometry of radar scans. This Bugfix now allows users to include or exclude based on the lower limit of reflectivity (variable called min_dbz can be set using the --skip command line option during data conversion). By default it is set to -35 dBZ and the special value of -99 is changed to be -34.5 for keeping those points for the output file. In practice, it appears there is never echo in the MRMS product below -30 dBZ so the special value keeps data in a tighter range than -99.

Issue(s) addressed

Resolves #1474

Dependencies

Should be no dependencies

Impact

Should be no impact to other repos.

Checklist

chengsiliu commented 4 months ago

I agree with the proposed code change.

I'd like to highlight a practical aspect concerning the direct assimilation of negative dBZ values using variational techniques. We've observed that including a significant number of negative dBZ values tends to slow down the convergence of the cost function significantly. Adjusting from 10 dBZ to -20 dBZ, compared to adjusting from 30 dBZ to 0 dBZ, results in a relatively minor change in the mixing ratio. Yet, it occupies a similar portion of the cost function adjustment effort. To ensure faster convergence and efficient iteration, we typically treat negative dBZ (except for missing value) values as zero dBZ in our preprocessing procedures in practice. This approach may compromise accuracy slightly, but it significantly enhances efficiency.

gthompsnJCSDA commented 4 months ago

@chengsiliu The timing of your review comment was impeccable. I was formulating an Email discussion to you when this comment arrived. Let's continue the discussion in Email. Obviously, we can permit whatever values we want to pass through a converter and deal with them differently in QC. That is ultimately a user decision in my opinion, and we should not be short-cutting the data because of past work if the future provides a means to discover something new and the code can operate the same regardless.

fabiolrdiniz commented 4 months ago

@gthompsnJCSDA, please, adjust your reference file if you are happy with the changes in the converter. The unit test is failing with:

ECCODES ERROR   :  Truncating time: non-zero seconds(41) ignored
DIFFER : LENGTHS OF RECORDS : Location (5419491) <> Location (249186)
DIFFER : LENGTHS : DIMENSION : Location : 5419491 <> 249186
DIFFER : VARIABLE : Location : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : height : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : latitude : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : longitude : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : dateTime : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : height : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : latitude : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : longitude : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
fabiolrdiniz commented 4 months ago

@gthompsnJCSDA, please, adjust your reference file if you are happy with the changes in the converter. The unit test is failing with:

ECCODES ERROR   :  Truncating time: non-zero seconds(41) ignored
DIFFER : LENGTHS OF RECORDS : Location (5419491) <> Location (249186)
DIFFER : LENGTHS : DIMENSION : Location : 5419491 <> 249186
DIFFER : VARIABLE : Location : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : height : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : latitude : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : longitude : ATTRIBUTE : _FillValue : VALUES : 9.96920997e+36 <> -999
DIFFER : VARIABLE : dateTime : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : height : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : latitude : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : longitude : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186
DIFFER : VARIABLE : reflectivity : SIZE : 5419491 <> 249186

@gthompsnJCSDA, this comment is valid still.

BenjaminRuston commented 4 months ago

@gthompsnJCSDA reference file still needs update looks like

gthompsnJCSDA commented 4 months ago

@gthompsnJCSDA reference file still needs update looks like

Fixed the testoutput file. Awaiting CI-tests to ensure all pass.

BenjaminRuston commented 4 months ago

@gthompsnJCSDA CI is not working properly do you think this one is ready otherwise

gthompsnJCSDA commented 4 months ago

@gthompsnJCSDA CI is not working properly do you think this one is ready otherwise

Someone needs to explain to me why the JEDI unit test fails, because I don't understand why.

PatNichols commented 4 months ago

@gthompsnJCSDA CI is not working properly do you think this one is ready otherwise

Someone needs to explain to me why the JEDI unit test fails, because I don't understand why.

The test did not run. There was an error setting up the CI to test the code. It's fixed now.