Ho-ChunHuang-NOAA
commented
4 years ago I discussed the GOES-16 ADP product content with NESDIS/aerosol validation group (Pubu Ciren) on September 18. The meaning of the Aerosol/Smoke/Dusk mask are different from my previous understanding.
The Aeorosl Mask is NOT an indicator of the AOD retrieval of "ALL" aerosol nor "Aerosol other than Smoke and Dust".
The Aerosol mask = 1 if [Smoke=1 OR Dust=1]. Thus there is no information of any pixel that has aerosol other than smoke and dust (such as anthropocentric aerosol) provided by ADP products. It is impossible to compute the fraction of Smoke AOD or Dust AOD over total AOD for each mapping grid.
NESDIS GOES-16 ADP product can only identify which ABI (sensor) pixel likely are all smoke/dust particle in its footprint, with confident level determined by the quality flag [qf] (bit 2-3 for smoke and bit 4-5 for dust). Since Smoke/Dust AOD is the final OBS (what we want) to be compared to same modeled variable, thus it may need to be read in by regrid_data_plane as additional constraint to find the AOD pixel to be gridded to selected model grid.
Note we already have total AOD qf to control the AOD mapping in input GOES-16 AOD file. This function should be retained because of the need to other broad community users. For NCEP operational need as a special case, to find the mapped Smoke AOD in model grid, we need to add additional condition of
High quality smoke AOD (-qc 0) = AOD with [high AOD qf]+[high Smoke confidence qf], i.e., total AOD qf <= 0 && smoke mask qf <=0.
Medium quality smoke AOD (-qc 1) = AOD with [high AOD qf]+[medium Smoke confidence qf] && [medium AOD qf]+[high Smoke confidence qf] && [medium AOD qf]+[medium Smoke confidence qf]. i.e., total AOD qf <= 1 && smoke mask qf <= 1 ??
And
Low quality Smoke AOD is total AOD qf <= 2 && smoke mask qf <= 2, ??
I do not know whether each pair occurred or not in reality, I need you expertise to refine it if you like to pursuit this approach.
I still want to use current regrid_data_plane function for total AOD comparison to that of NAQFC/CMAQ or FV3GFS-CHEM (or GEFS-aerosol).
For the Smoke AOD mapping I would like to create to be compared to HYSPLIT/HRRR-smoke smoke AOD output
Using -qc 0, -qc 0,1 and -qc 0,1,2.
I attached a new ADP description from NESDIS USer's Guide here
GOES-16 ADP Users Guide extract.pdf
Full version of the users guide can be found in IDL readme docs in John's comment.
JohnHalleyGotway
hsoh-u
This is a request from the NOAA/EMC Aerosols group. The regrid_data_plane tool was enhanced to process GOES-16 AOD data. During a meeting at NCWCP on 9/11/2019, Ho-Chun Huang requested that regrid_data_plane be enhanced to also process data from GOES-16 ADP files.
See a description of the GOES-16 ADP data format attached. GOES-16_ADP_readme_idl_code_20180328.docx
Also see and excerpt from the ADP user's guide attached. GOES-16 ADP Users Guide extract.pdf
The ADP files are very similar to the AOD files, but slightly different. The relevant data variables are named Aerosol, Dust, and Smoke. Each of these fields contains 0's or 1's as a mask to define whether Aerosol, Dust, or Smoke is present at that pixel. In addition, the DQF variable contains quality control flags. However, the QC info is stored in the 8 bits of the DQF values as described in the document above.
When processing Smoke or Dust, parse the DCF bits to construct a QC value of 0, 1, or 2, for high, medium, and low quality, respectively. Use the user-specified -qc flag command line option to determine which pixels should be used. When doing the regridding step, determine how many of the pixels inside each grid box meet the -qc flag setting. And then report the output value for that grid box as the average value of the good data pixel values. The result should be a number between 0 and 1 indicating the percentage of the pixels in that grid box containing smoke or dust.
Ultimately, Ho-Chun plans to process both the ADP files to compute fields for smoke percentage and dust percentage. Then he'll use those percentages to define masks when computing stats in Grid-Stat for the GOES-16 AOD values. For example, he can compute the RMSE for AOD for grid boxes that are at least 50% smoke.
Even though Ho-Chun has no planned usage for the Aerosol field, we may as well include that as an optional output.
Here's a description of how the -valid_thresh command line option of regrid_data_plane should be used. Let's say one target output grid box has 100 GOES-16 pixels inside of it. The -valid_thresh option defines the ratio (between 0 and 1) of those 100 pixels which must meet the requested -qc flag information in order to get an output value for that grid box.
Sample data can be found on WCOSS, here: The daily ADP data file can be found in (phase II) /gpfs/gd3/emc/meso/noscrub/Ho-Chun.Huang/GOES16_AOD/ADP/yyyymmdd
module load NetCDF/4.2/serial
You can use ncdump to read the header, I believe we will read Smoke, Dust, and DQF (for high, medium, and low confidence of Smoke and Dust mask)