AWIPS CLAVRx file issues

[kathys]

I spent some time comparing the CLAVRx products created using P2G Version 3.0 versus the ones created with Version 2.3. I was able to compare SNPP and NOAA20 VIIRS and Aqua and Terra MODIS products. Almost all of the files displayed very, very closely between v3.0 and v2.3 with two exceptions.

1). The P2G Version 3.0 effective radius products (cld_reff - both dcomp and nlcomp) showed much higher values when using the AWIPS readout. For example, the Terra MODIS dcomp cld reff readout showed at one pixel for Version 3.0 a value of 16016 while at the sample pixel a value of 7964 with the Version 2.3 output.

2). The cloud phase product displays as all blue in AWIPS using Version 3.0 output, while the Version 2.3 product display looks much different. See images below. This was the case for all 4 instruments that I examined.

3). The clavrx filtering for day/night does not seem to be working correctly. I am using an NPP nighttime VIIRS CLAVRx dataset, and it will not create the default files cld_opd_nlcomp cld_reff_nlcomp refl_lunar_dnb_nom files with a message in the log file that says polar2grid.filters._base : filter_scene : Unloading 'DataID(name='refl_lunar_dnb_nom', resolution=742, modifiers=())' because there is not enough day data." The log files is here ->bumi:/data/users/kathys/test_data/clavrx`

[kathys]

As an FYI I added number 3 above. The day/night filtering for P2G does not appear to be working correctly, at least for nighttime datasets.

[djhoese]

those likely need to be special cased. The default rule is to check anything calling itself "reflectance" for daytime coverage. These obviously need to be excluded or at least only done for night checks.

[kathys]

Here is the difference between the MODIS CLAVRx AWIPS NetCDF file Cloud Phase product header between P2G Version 2.3 and Version 3.0. Not sure what the problem is.

V2.3: short data(y, x) ; data:_FillValue = 32767s ; data:coordinates = "y x" ; data:scale_factor = 0.5f ; data:add_offset = 0.f ; data:units = "*1" ; data:valid_min = 0s ; data:valid_max = 32766s ; data:standard_name = "cloud_phase" ; data:grid_mapping = "lambert_projection" ;

V3.0: ushort data(y, x) ; data:_FillValue = 65409US ; data:units = "*1" ; data:physical_element = "CLAVR-x Cloud Phase" ; data:grid_mapping = "lambert_projection" ;

[djhoese]

Here is using pure satpy remapping the aqua modis cloud phase that was used in the above AWIPS screenshots. It looks like the very yellow image is actually correct for the data. The problem is that the AWIPS colormap makes both 0 and 1 values black so it takes away from the "dynamic" view of the data. Additionally, I don't see any 3 (Mixed) values in the data. So now we know that this is correct for the data we have I can start debugging how to make it consistent. If I remember correctly the cloud_phase changed when I forced the scale factor and add offset, but the cloud_type got worse. That's up next.

For reference/history, here is cloud_type:

[djhoese]

Kathy and I met yesterday and tried out a few AWIPS things to figure out what is going on with some of these products. Here are our findings and things we need to do:

1. The styleRules at SSEC are wrong for the CLAVR-x Cloud Type product for the product produced by P2G v2.3 (see /awips2/edex/data/utility/common_static/site/OAX/styleRules/polarScmiImageryStyleRules.xml on awipsdata6). The category mapping increments by 1 when it should increment by 2 to match the 0.5/0 factor/offset used by the data.
2. AWIPS can handle category products with no scale_factor or add_offset attributes. In v2.3 we did 0.5 and 0 to make AWIPS happy. Now, the only reason to keep doing it is to match existing styleRules that expect the 0.5/0 scaling.
3. The styleRules for the category products for some reason list 65535 as the fill value. This is not the fill value used by these products and I'm not sure why it is forced/specified at all (can't AWIPS default to _FillValue of the NetCDF file?).
4. Cloud Effective Radius has a unit of "microm" in P2G and was explicitly converted that way from "micron" so it would seem at some point this worked. However, AWIPS' styleRules for this product do not list a "displayUnits" and end up displaying the in-file scaled values instead of the micron values. We tried "microm", "micron", and "1" when testing. Only "1" had AWIPS scale the values from the in-file scale_factor/add_offset.
5. For some reason in later testing, "Cloud Type" kept saying it was a duplicate record for every time even though it was new. It never showed up in the client after initial test files.

TODO:

1. @djhoese: @kathys I realized I should try "um" for cloud effective radius units. Or "m-6" or something more udunits friendly.
2. @kathys: Email Carl/@spruceboy for the styleRules they use to see if they have an fixes or changes from what we're seeing on our AWIPS server. For example, the colormap/mapping of cloud type.
3. @djhoese: Write a test script to produce test AWIPS SCMI files with a gradient of data across them. The script will let you control data type, _Unsigned special attribute, physical element, etc.
4. @djhoese: I will email Lee Cronce to see what knowledge he (or his contacts) have about when styleRules unit conversion is done.

[spruceboy]

I will ask Carl and see if he can get you guys the configs he provides the Alaska region NWS for AWIPS. Thanks!

[djhoese]

Looks like I forgot to update this issue from prior news and I have new news, so here we go:

1. @kathys pointed out in the original description that cld_reff_dcomp/nlcomp products are showing much higher values than they should. This was discovered to be because the AWIPS-friendly units being provided ("microm") were not actually parse-able by AWIPS. This caused AWIPS to fallback to showing the scaled in-file values which are large 16-bit integers. Today Lee Cronce and I sat down and sifted through the AWIPS source code. We discovered that from AWIPS 19 to AWIPS 20 the AWIPS team switched from using javax.measure for unit conversion to the uom-se library. The version of uom-se bundled with AWIPS is relatively old and has a huge bug where any micro- prefixed version of a symbol does not exist. Lee Cronce is going to contact some AWIPS folks and see what it would take to update uom-se in the next version of AWIPS (likely in 9-24 months). So bottom line, this is a bug in AWIPS and there isn't really a way I see to get around it.
2. The cloud type and cloud phase products weren't displaying correctly because the colormaps being applied to them in AWIPS were based on the old P2G's way of scaling the products. P2G v2.3 would multiple the values by 2 because AWIPS was not happy about a scale factor of 1. This is no longer needed functionally but is required for backwards compatibility with the old colormap configuration (styleRules in AWIPS). Satpy has been updated to reflect this.

[djhoese]

@kathys for the day/night filtering, are the dcomp products created even when it is a completely night scene? Based on the NPP VIIRS files you gave me it looks like yes? I'm wondering if I could just skip computing the day/night coverage and just assume that if the variable exists then the product can be made.

[kathys]

Unfortunately, it looks like the product arrays are all there for all of the VIIRS files, day or night. There is a dcomp_info array which includes a day/night and a twilight mask, and an nlcomp_info flag too. @nbearson does clavrx quicklooks for the CSPP release, and he has had to to this. I will ask him about it.

[djhoese]

Not a big deal. It was a small optimization that would be possible. I'll fix the filtering.

[djhoese]

Now with #541 merged these issues should all be fixed except for the AWIPS "microm" bug.