Issues with polar2grid and AVHRR clavrx products..

[spruceboy]

I am trying to make geotifs and scmi tiles from AVHRR clavrx products, and am running into problems. Maybe I am doing it wrong?

With the lastest version of polar2grid i have (2.3.0a2), it generates an error message.

[jecable@roundrect tst]$ ./polar2grid-swbundle-2.3.0a2/bin/polar2grid.sh  clavrx gtiff  -g 203 -p cld_temp_acha -f ./in/clavrx_hrpt_noaa19_20191031_1910_55298.l1b.level2.hdf
INFO     : Initializing reader...
ERROR    : clavrx frontend failed to load and sort data files (see log for details)

I have attached the log file.

clavrx2gtiff_fail.log

Using an older version of polar2grid (2.2.1) it seems to just run forever:

./polar2grid_v_2_2_1/bin/polar2grid.sh  clavrx gtiff  -g 203 -p cld_temp_acha -f ./in/clavrx_hrpt_noaa19_20191031_1910_55298.l1b.level2.hdf
INFO     : Initializing reader...
INFO     : Initializing remapping...
INFO     : Initializing backend...
INFO     : Initializing compositor objects...
INFO     : Extracting swaths from data files available...
/home/jecable/tst/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/pyresample/geometry.py:86: UserWarning: All geometry objects expect longitudes in the [-180:+180[ range. We will now automatically wrap your longitudes into [-180:+180[, and continue. To avoid this warning next time, use routine utils.wrap_longitudes().
  'To avoid this warning next time, use routine utils.wrap_longitudes().')
INFO     : Remapping to grid 203
INFO     : Grid information:
        grid_name: 203
        proj4_definition: +proj=stere +a=6371200 +b=6371200 +lat_0=90 +lat_ts=60.0 +lon_0=-150 +units=m +no_defs
        cell_width: 1000.0
        cell_height: -1000.0
        width: 8384
        height: 7239
        origin_x: -4952960.85604
        origin_y: 381022.610378
WARNING  : Data has less than 2 rows per scan, this is not optimal for the EWA resampling algorithm. All rows will be used as one scan
..

I am probibly doing something wrong, if so please let me know what i am missing.

The file I am using for testing is here: http://hippy.gina.alaska.edu/distro/clavrx_hrpt_noaa19_20191031_1910_55298.l1b.level2.hdf

Thanks again for the great tool!

[djhoese]

This file is from a new version of clavrx right? It looks like the Satpy reader for CLAVR-x can't handle their changes in the sensor global attribute (changed from "AVHRR" -> "AVHRR-3"). Let me see if I can fix Satpy to work with the file you've provided and give you a workaround if I can get it to work.

[djhoese]

I was able to fix the Satpy part of this here https://github.com/pytroll/satpy/pull/969

I've also solved another bug in #237.

As for the hanging remapping, that is still a problem that I'm working on. Do you know if this ever performed well in the past? Not like 30 minutes long, but a couple minutes? This has to do with the EWA resampling algorithm wanting to process one scan at a time, but since AVHRR doesn't have scans we've been cheating and processing the entire swath at once. For some reason this now no longer works. It isn't that it is stuck, but rather it just takes a really really long time to get through the processing. A "simple" workaround might be to switch to nearest neighbor resampling, but we'll have to verify the quality of the output.

I'm still working on this. @kathys and I will have to talk about our options next time we meet.

[spruceboy]

Thanks for taking a look at it, I will try those fixes next week when i am in the office. I don't know if the avhrr clavrx products ever worked well, we are only now starting to provide them to the NWS in any real form.

[djhoese]

This issue should be affecting regular AVHRR L1b files too. Are you using those at all? Have you seen these issues?

I'll have to double check that I was seeing it with the L1b next time I test this.

[djhoese]

Ok this is a little strange. I tried it with an AVHRR L1b file (hrpt_noaa19_20140731_1007_28224.l1b) and it ran relatively fast. This is odd since both the L1b files and the clavrx files should be using the same remapping method (relatively). The only difference is the actual data being resampled from what I can tell. My next best guess is the number of NaNs in the data (in the file you sent me rain_rate is all NaNs) and I wonder if these are being processed extremely slow. I guess I'll have to do even more playing around.

[spruceboy]

We make SCMI and geotifs from AVHRR L1b from NOAA POES and Metop AVHRR and don't have any issues. It runs fast and works great. (Sorry I didn't reply earlier!)

We actually don't care about the rain rates. I can give you a list of the products that folks are actually using, it is pretty small, if that helps at all.

[spruceboy]

For geotifs, we use data products: cld_emiss_acha cld_height_acha cld_opd_dcomp cld_reff_dcomp cld_temp_acha cloud_phase cloud_type cld_base_altitude_acha ccl_layer_flag ccl_1 ccl_2 ccl_3 ccl_4 ccl_5

For SCMI we make these data products: cld_emiss_acha cld_height_acha cld_opd_dcomp cld_reff_dcomp cld_temp_acha cloud_phase cloud_type cld_height_base_acha rain_rate

So I guess we are making the rain rate, but I don't think anyone is using them (they don't appear to reflect reality well).

Thanks again for taking a look at this!

[djhoese]

Thanks for the information. I'll focus on one of the other products, but based on my tests it doesn't seem to matter. This is such a strange problem.

[djhoese]

I found the problem! The first 8 rows of the longitude and latitude data in the clavrx data are all NaNs. There must be something in the resampling code that just can't handle this. If I force these NaN values to anything realistic then the resampling works fine.

I'll look at this more next week. If you or Carl or someone else knows why the navigation is all fill values for these rows let me know. I'll ask some people at SSEC about it.

[spruceboy]

Hmm, I wonder if I just picked a bad example then. I did try several passes, and they all hung.
The geotifs from the level1 data look funky, maybe there is something wrong with the data.

[spruceboy]

The original data (raw, and level1b) can be found here: http://hippy.gina.alaska.edu/distro/strange_avhrr/

Thanks again for taking a look at this!

[spruceboy]

Hmm, there is a bunch of striping we don't normally see in the that data -> http://hippy.gina.alaska.edu/distro/strange_avhrr/noaa19_avhrr_band4_bt_20191031_191037_203_1k.png

[djhoese]

I have some data from Liam Gumley's processing and I saw the same behavior (hanging/long processing). I'm asking around to see if this is expected from clavrx. However, if that's what the l1b files look like then maybe it isn't clavrx's fault necessarily.

[djhoese]

I just sent you an email with the newest beta tarball for Polar2Grid 2.3.0. It should include all of the necessary fixes for the things you've pointed out. When you've tested things and they work we can close these issues.

[spruceboy]

Thanks, I really appreciate it!

[djhoese]

@spruceboy Just wanted to clarify something after talking with @wstraka3 about this: have you always processed AVHRR clavrx data with Polar2Grid or did you just start doing it since this latest clavrx update?

Edit: If you have any logs for the AVHRR processing (clavrx logs most importantly, but others along the chain too) it would be greatly appreciated.

[spruceboy]

We have only just started processing avhrr clavrx data - this problem came up with my first attempt at it. We do regularly make scmi and geotifs from the L1b (so imagery) however. Hopefully I didn't confuse / misstate that earlier, sorry for any confusion.

[wstraka3]

@spruceboy - Having worked with HRPT data for 10 years or so in CLAVR-x, I have noticed that often at the beginning of the pass there are several missing lines. The way the code is set up, CLAVR-x would fill those in as missing values. So, this isn't an uncommon occurrence. As @djhoese mentioned, a CLAVR-x processing log file from just one of the runs would be helpful.

[djhoese]

@spruceboy Have you processed the L1b data in the past? Have you ever seen these types of issues?

@kathys Are you processing AVHRR L1b data with Polar2Grid? Is Liam?

If Polar2Grid has been used in the past to process AVHRR data from direct broadcast and it has included "bad" or "missing" geolocation information at the beginning of the pass then Polar2Grid should have been failing already. Since I haven't heard of this issue before then either people haven't been using it on direct broadcast data or their is a bug in the L1b processing that is changing the values of the geolocation to be "good enough" that they don't cause the EWA resampling to mess up so terribly the way it is with the clavrx output.

Now that I think about it, the AVHRR L1b reader in Polar2Grid has to interpolate/extrapolate the geolocation. I should plot the before and after of that. That might explain some of this. I'll look at that tomorrow.

[spruceboy]

@djhoese - We have regularrly processed L1b (from AAPP) from NOAA Poes for at least a year now. The Alaska region NWS has been using it and hasn't complained about any issues. I think we are providing SCMI and regional sat products.

@wstraka3 - I can get you some log files from the CLAVRX runs from NOAA Poes AVHRR when I am back in the office Tuesday.

Thanks again for looking at this!

[wstraka3]

@spruceboy - So, I looked into this. The basic issue is that you are likely using CLAVR-x such that it is using the operational (onboard) thermal coefficients. Even though there is data in the L1b data, the coefficients require a few lines of spinup. I won't do int the detail here, but suffice to say this is why you are seeing those "missing" geolocations in the first few scans on an HRPT pass.

There are two options:

• Polar2grid is modified to accommodate the fact the calibration needs to spin up

• You use the recalibrated coefficients using PATMOS-x numbers. This would mean you would not using the operational calibration. This would be a simple change in your default options file for CLAVR-x

Trying to modify CLAVR-x to fix just this as well as ensure the other sensors (LEO and GEO) work would be a pretty significant undertaking for something as relatively small as this.

[kathys]

@wstraka3 Is this something new that has been added in recent versions of CLAVRx? I don't remember this being a problem in older versions. When did it start using onboard thermal coefficients?

[wstraka3]

@kathys , since the code controlling which one is used has been consistent for quite a while (i.e. back to the old CVS repo), my best guess, without seeing the options file, is that somehow got the thermal cal option flipped from 1 to 0. We'd need to look at the older and current options file to be sure.

[spruceboy]

We are using a totally stock options file for AVHRR in CLAVRX 3.0 . If you not already found it, it appears to be generated by the "mk_avhrr_options.sh" script, which I have put below. If you tell me what options to change, I try it.

#/usr/bin/env sh

echo "${CLAVRX_ANC}/
${TEMP_DIR}/
2   !Expert mode (0 - nothing is read below, 1 - only !E1 and lower is read, 9 - everything is read )
5   !E1 Messaging Control Flag (QUIET = 0,ERROR = 1,MINIMAL = 2,WARNING = 4,DEFAULT = 5,VERBOSE = 9)
1   !E1 ALG Mask mode bayesian cloud mask (0=no, 1 = yes)
9   !E1 ALG DCOMP (0=off,1=0.65/1.6, 2=0.65/2.2, 3=0.65/3.75, 9 = 0.65/1.6/2.2/3.75)
default   !E1 ALG ACHA (0=off,1=11,2=11/6.7,3=11/12,4=11/13.3,5=11/8.5/12,6=11/6.7/12,7=11/6.7/13.3/,8=11/12/13.3)
2   !E1 ALG CCL Mode (0 =off,1=top only,2 = top + base,3 = top + base + lower)
0   !E1 ALG CCL Type = (0=NOAT, 1=ISSCP, 2=NCEP)
0   !E1 ALG ASOS Mode (0=off, 1=on)
0   !E1 ALG NLCOMP (0=off,1=on)
0   !E1 ALG Aerosol (0=off,1=on)
1   !E2 OUTPUT Level-2  file output flag (0= no, 1 = yes)
0   !E2 OUTPUT format flag (0= hdf4, 1 = netcdf4)
1   !E2 PRC Cloud flag
500 !E2 Num scan lines
0   !E2 SASRAB switch on/off SASRAB parameters
${NWP_FLAG}   !E2 NWP nwp option  (1=gfs,2=ncep reanalysis,3=cfsr)
0   !E2 NWP Nwp Mode  (0=minimal, 1=all)
1   !E2 RTM rtm option  (0=crtm,1=pfast, (2=rttov))
0   !E2 NAV nav option  (0 =  Use  level-1b,1 = external (future) 2 = Reposnx ( fred nagle): For non-AVHRR, use 0)
1   !E2 OUT output compression flag (0=no,1=gzip)
0   !E2 MASK read auxilary cloud mask 1b (1 = read from 1b and don't recompute)
default
1   !E3 SFC seebor emiss flag (0=no, 1=yes)
1   !E3 SFC sea emiss flag (0=no, 1=yes)
0   !E3 SFC read hires sfc type flag (0=no-8km, 1 = yes-1km)
1   !E3 SFC read land mask flag (0=no, 1=yes) (goge2_0ll.hdf)
1   !E3 SFC read coast mask flag (0=no, 1=yes) (coast_mask_xkm.hdf, x = 1 or 8)
1   !E3 SFC read surface elevation flag (0=no, 1=yes) (GLOBE_xkm_digelev.hdf, x = 1 or 8)
0   !E3 SFC read volcano mask flag (0=no, 1=yes) (volcano_mask_1km.hdf)
1   !E3 SFC read snow mask flag (0=no, 1 = ims, 2 = GlobSnow)
0   !E3 SFC read dark composite flag (0=no, 1 = yes) only GEO
0   !E4 AVHRR-ONLY specific ref_cal_1b flag (0 = use default level1b cal, 1 = recalibrate using PATMOS-x numbers (AVHRR-only))
0   !E4 AVHRR-ONLY therm_cal_1b flag (0 = use defauly level1b cal, 1= recalibrate using PATMOS-x numbers (AVHRR-only))
1   !E5 MASK lrc_flag (0=no,1=yes)
1   !E5 NWP smooth nwp flag (0=no, 1=yes)
0   !E5 process_undetected_flag (0=no,1=yes) ( means process all pixels cloudy and cloud-free
0 30.0 60.0 0.0 180.0 0.0 90.0 0.0 180.0 ZONE2    !E6 on_flag Lat_South,Lat_North,Lon_West,Lon_East,Zen_Min,Zen_Max,Solzen_Min,Solzen_Max, Name
0 1 1 !E6 native res sample(0)/average(1), X_stride(>=1), Y_Stride (>=1)
1 1 1 1 1 1    !E6 chan on flags of channels 1,2,3,4,5,6
1 1 1 1 1 1    !E6 chan on flags of channels 7,8,9,10,11,12
1 1 1 1 1 1    !E6 chan on flags of channels 13,14,15,16,17,18
1 1 1 1 1 1    !E6 chan on flags of channels 19,20,21,22,23,24
1 1 1 1 1 1    !E6 chan on flags of channels 25,26,27,28,29,30
1 1 1 1 1 1    !E6 chan on flags of channels 31,32,33,34,35,36
0 0 0 0 0 0    !E6 chan on flags of channels 37(AHI-only), 38(AHI-only), 39(I1), 40(I2), 41(I3), 42(I4)
0 0 0 0 0 0    !E6 chan on flags  of channels 43(I5), 44(DNB), 45-48 (Spare)"

[spruceboy]

Here is the version 2 script:

echo "${CLAVRX_ANC}/
${TEMP_DIR}/
4   !Expert mode (0 - nothing is read below, 1 - only !E1 and lower is read, 9 - everything is read )
1   !E1 ALG Mask mode bayesian cloud mask (0=no, 1 = yes)
3   !E1 ALG DCOMP
3   !E1 ALG ACHA
1   !E1 ALG NLCOMP
1   !E2 OUTPUT Level-2  file output flag (0= no, 1 = yes)
0   !E2 OUTPUT RTM  file output flag (0= no, 1 = yes)
1   !E2 PRC Cloud flag
500 !E2 Num scan lines
0   !E2 SASRAB switch on/off SASRAB parameters
${NWP_FLAG}   !E2 NWP nwp option  (1=gfs,2=ncep reanalysis,3=cfsr,4=gdas)
1   !E2 RTM rtm option  (0=crtm,1=pfast, (2=rttov))
0   !E2 NAV nav option  (0 =  Use  level-1b,1 = external (future) 2 = Reposnx ( fred nagle): For non-AVHRR, use 0)
1   !E2 OUT output compression flag (0=no,1=gzip)
0   !E2 MASK read auxilary cloud mask 1b (1 = read from 1b and don't recompute)
avhrr_default_nb_cloud_mask_lut.nc
1   !E3 SFC seebor emiss flag (0=no, 1=yes)
0   !E3 SFC read hires sfc type flag (0=no-8km, 1 = yes-1km)
1   !E3 SFC read land mask flag (0=no, 1=yes) (goge2_0ll.hdf)
1   !E3 SFC read coast mask flag (0=no, 1=yes) (coast_mask_xkm.hdf, x = 1 or 8)
1   !E3 SFC read surface elevation flag (0=no, 1=yes) (GLOBE_xkm_digelev.hdf, x = 1 or 8)
0   !E3 SFC read volcano mask flag (0=no, 1=yes) (volcano_mask_1km.hdf)
1   !E3 SFC read snow mask flag (0=no, 1 = ims, 2 = GlobSnow)
0   !E3 SFC read dark composite flag (0=no, 1 = yes) only GEO
1   !E4 AVHRR-ONLY specific ref_cal_1b flag (0 = use default level1b cal, 1 = recalibrate using PATMOS-x numbers (AVHRR-only))
1   !E4 AVHRR-ONLY therm_cal_1b flag (0 = use defauly level1b cal, 1= recalibrate using PATMOS-x numbers (AVHRR-only))
1   !E5 MASK lrc_flag (0=no,1=yes)
1   !E5 NWP smooth nwp flag (0=no, 1=yes)
0   !E5 process_undetected_flag (0=no,1=yes) ( means process all pixels cloudy and cloud-free
0 90.0 -90.0 180.0 -180.0 85.0 0.0 180.0 0.0    !E6 on_flag Lat_Max,Lat_Min,Lon_Max,Lon_Min,Zen_Max,Zen_Min,Solzen_Max,Solzen_Min
1 1 1 1 1 1    !E6 chan on flags of channels 1,2,3,4,5,6
1 1 1 1 1 1    !E6 chan on flags of channels 7,8,9,10,11,12
1 1 1 1 1 1    !E6 chan on flags of channels 13,14,15,16,17,18
1 1 1 1 1 1    !E6 chan on flags of channels 19,20,21,22,23,24
1 1 1 1 1 1    !E6 chan on flags of channels 25,26,27,28,29,30
1 1 1 1 1 1    !E6 chan on flags of channels 31,32,33,34,35,36
0 0 0 0 0 0    !E6 chan on flags of channels 37(AHI-only), 38(AHI-only), 39(I1), 40(I2), 41(I3), 42(I4)
0 0 0 0 0 0    !E6 chan on flags of channels 43(I5), 44(DNB), 45-48 (Spare)"

[spruceboy]

It looks like these lines have changed - are these the calibration parameters you are refering to @wstraka3 ?

29,30c36,37
< 1   !E4 AVHRR-ONLY specific ref_cal_1b flag (0 = use default level1b cal, 1 = recalibrate using PATMOS-x numbers (AVHRR-only))
< 1   !E4 AVHRR-ONLY therm_cal_1b flag (0 = use defauly level1b cal, 1= recalibrate using PATMOS-x numbers (AVHRR-only))
---
> 0   !E4 AVHRR-ONLY specific ref_cal_1b flag (0 = use default level1b cal, 1 = recalibrate using PATMOS-x numbers (AVHRR-only))
> 0   !E4 AVHRR-ONLY therm_cal_1b flag (0 = use defauly level1b cal, 1= recalibrate using PATMOS-x numbers (AVHRR-only))

Full diff:

[jecable@nrt-cloud-test-01 cspp]$ diff CLAVRx_2_0/options/mk_avhrr_options.sh CLAVRx_3_0/options/mk_avhrr_options.sh 
5c5,6
< 4   !Expert mode (0 - nothing is read below, 1 - only !E1 and lower is read, 9 - everything is read )
---
> 2   !Expert mode (0 - nothing is read below, 1 - only !E1 and lower is read, 9 - everything is read )
> 5   !E1 Messaging Control Flag (QUIET = 0,ERROR = 1,MINIMAL = 2,WARNING = 4,DEFAULT = 5,VERBOSE = 9)
7,9c8,14
< 3   !E1 ALG DCOMP
< 3   !E1 ALG ACHA
< 1   !E1 ALG NLCOMP
---
> 9   !E1 ALG DCOMP (0=off,1=0.65/1.6, 2=0.65/2.2, 3=0.65/3.75, 9 = 0.65/1.6/2.2/3.75)
> default   !E1 ALG ACHA (0=off,1=11,2=11/6.7,3=11/12,4=11/13.3,5=11/8.5/12,6=11/6.7/12,7=11/6.7/13.3/,8=11/12/13.3)
> 2   !E1 ALG CCL Mode (0 =off,1=top only,2 = top + base,3 = top + base + lower)
> 0   !E1 ALG CCL Type = (0=NOAT, 1=ISSCP, 2=NCEP)
> 0   !E1 ALG ASOS Mode (0=off, 1=on)
> 0   !E1 ALG NLCOMP (0=off,1=on)
> 0   !E1 ALG Aerosol (0=off,1=on)
11c16
< 0   !E2 OUTPUT RTM  file output flag (0= no, 1 = yes)
---
> 0   !E2 OUTPUT format flag (0= hdf4, 1 = netcdf4)
15c20,21
< ${NWP_FLAG}   !E2 NWP nwp option  (1=gfs,2=ncep reanalysis,3=cfsr,4=gdas)
---
> ${NWP_FLAG}   !E2 NWP nwp option  (1=gfs,2=ncep reanalysis,3=cfsr)
> 0   !E2 NWP Nwp Mode  (0=minimal, 1=all)
20c26
< avhrr_default_nb_cloud_mask_lut.nc
---
> default
21a28
> 1   !E3 SFC sea emiss flag (0=no, 1=yes)
29,30c36,37
< 1   !E4 AVHRR-ONLY specific ref_cal_1b flag (0 = use default level1b cal, 1 = recalibrate using PATMOS-x numbers (AVHRR-only))
< 1   !E4 AVHRR-ONLY therm_cal_1b flag (0 = use defauly level1b cal, 1= recalibrate using PATMOS-x numbers (AVHRR-only))
---
> 0   !E4 AVHRR-ONLY specific ref_cal_1b flag (0 = use default level1b cal, 1 = recalibrate using PATMOS-x numbers (AVHRR-only))
> 0   !E4 AVHRR-ONLY therm_cal_1b flag (0 = use defauly level1b cal, 1= recalibrate using PATMOS-x numbers (AVHRR-only))
34c41,42
< 0 90.0 -90.0 180.0 -180.0 85.0 0.0 180.0 0.0    !E6 on_flag Lat_Max,Lat_Min,Lon_Max,Lon_Min,Zen_Max,Zen_Min,Solzen_Max,Solzen_Min
---
> 0 30.0 60.0 0.0 180.0 0.0 90.0 0.0 180.0 ZONE2    !E6 on_flag Lat_South,Lat_North,Lon_West,Lon_East,Zen_Min,Zen_Max,Solzen_Min,Solzen_Max, Name
> 0 1 1 !E6 native res sample(0)/average(1), X_stride(>=1), Y_Stride (>=1)
42c50
< 0 0 0 0 0 0    !E6 chan on flags of channels 43(I5), 44(DNB), 45-48 (Spare)"
---
> 0 0 0 0 0 0    !E6 chan on flags  of channels 43(I5), 44(DNB), 45-48 (Spare)"

[wstraka3]

@spruceboy - Yep, that is exactly what I was talking about. You'd just need to flip it back to 1 if you don't want the missing lines at the beginning of the scan.

@kathys - Based on the SVN tag for that delivery, I am guessing someone forgot to flip for you guys in the delivery to CSPP. Not sure about the whole delivery process for CSPP, since I'm not involved with that.

[spruceboy]

Just to follow up inside the issue so there is a record - the lastest polar2grid release that @djhoese made for us works great. Changing the options so these are set didn't though, at least with the older versions of polar2grid - perhaps other values in the options file also needs to be tweeked in the options file for them to take effect : 1 !E4 AVHRR-ONLY specific ref_cal_1b flag (0 = use default level1b cal, 1 = recalibrate using PATMOS-x numbers (AVHRR-only)) 1 !E4 AVHRR-ONLY therm_cal_1b flag (0 = use defauly level1b cal, 1= recalibrate using PATMOS-x numbers (AVHRR-only))

We are going to follow up via email about some other issues with the NOAA Poes AVHRR products - they are looking pretty strange, but we think polar2grid is working fine at this point.

Sorry we didn't follow up faster, no sooner than I raise this issue and our main AVHRR feed went out due to a series of rolling power outages at that site that they still have not entirely recovered from.

Thanks again!

[djhoese]

Thanks for the update. Closing this now as it seems we've done all we can do on the P2G side.