search

NOAA-EMC / GDASApp

Global Data Assimilation System Application
GNU Lesser General Public License v2.1
14 stars 30 forks source link

Add AVHRR/NOAA-15/18/19 assimilation to end-to-end GDASApp validation #997

Closed BrettHoover-NOAA closed 5 months ago

BrettHoover-NOAA commented 5 months ago

Adding satwinds from the Advanced Very High Resolution Radiometer (AVHRR) from NOAA-15/18/19 to GDASApp end-to-end testing

new files include: parm/atm/obs/config/satwind_avhrr_n15.yaml.j2: QC filter YAML for AVHRR NOAA-15 satwinds (jinja2 standard) parm/atm/obs/config/satwind_avhrr_n18.yaml.j2: QC filter YAML for AVHRR NOAA-18 satwinds (jinja2 standard) parm/atm/obs/config/satwind_avhrr_n19.yaml.j2: QC filter YAML for AVHRR NOAA-19 satwinds (jinja2 standard) parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_avhrr.json: JSON containing data format, sensor, and satellite information for AVHRR NOAA-15/18/19 satwinds ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_avhrr.py: bufr2ioda code for extracting AVHRR NOAA-15/18/19 satwinds from BUFR

See #996 for testing results. JEDI/GSI comparisons look good with GSI thinning turned off, but there are large ob-count disparities both before and after QC when comparing GSI+thinning to JEDI+thinning.

BrettHoover-NOAA commented 5 months ago

@RussTreadon-NOAA - looks like your approval was automatically dismissed after I submitted a small update, may need it again

emilyhcliu commented 4 months ago

@BrettHoover-NOAA I have two questions for AVHRR winds. For 2021080100, we also have AVHRR AMVs from MetOP-A and MetOp-B. I dumped out the satwnd bufr file and found that we have SAIDs: 206, 209, 233, 3, 4 (they are NOAA-15, NOAA-18, NOAA-19, MetOp-A, and MetOp-B, respectively) Do you plan to add AVHRR from MetOp satellites later?

Here is the convinfo valid for 2021080100

 uv       244    3    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244    4    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244    5    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  206    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  207    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  209    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  223    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8

For current operational, we do not assimilate AVHRR from NOAA satellites anymore. But it is still registrated in the convinfo file. It is a bit odd. I see AVHRR winds from NOAA-15, 18, and 19 in the satwnd bufr dump file (NC005080). The MetOp-B and MetOp-C are in the same dump file under NC005081.
Is there any reason we do not use AVHRR winds from NOAA satellites anymore? (we are still using AVHRR radiance data from NOAA satellites NOAA-19 and NOAA-18).

Current OPS (v16.3.14)

 uv       244    3    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244    4    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244    5    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  206    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  207    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  209    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  223    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8

Thanks!!

emilyhcliu commented 4 months ago

@BrettHoover-NOAA I have two questions for AVHRR winds. For 2021080100, we also have AVHRR AMVs from MetOP-A and MetOp-B. I dumped out the satwnd bufr file and found that we have SAIDs: 206, 209, 233, 3, 4 (they are NOAA-15, NOAA-18, NOAA-19, MetOp-A, and MetOp-B, respectively) Do you plan to add AVHRR from MetOp satellites later?

Here is the convinfo valid for 2021080100

 uv       244    3    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244    4    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244    5    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  206    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  207    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  209    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  223    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8

For current operational, we do not assimilate AVHRR from NOAA satellites anymore. But it is still registrated in the convinfo file. It is a bit odd. I see AVHRR winds from NOAA-15, 18, and 19 in the satwnd bufr dump file (NC005080). The MetOp-B and MetOp-C are in the same dump file under NC005081. Is there any reason we do not use AVHRR winds from NOAA satellites anymore? (we are still using AVHRR radiance data from NOAA satellites NOAA-19 and NOAA-18).

Current OPS (v16.3.14)

 uv       244    3    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244    4    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244    5    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  206    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  207    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  209    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8
 uv       244  223    1     3.0      0      0      0   2.5  20.0   1.4   2.5  0.005000     0    0.     0.      0    0.     0.   15   -8

Thanks!!

Oh, I see. I checked the NC005080 (NOAA AVHRR winds) in the current operational run. There are very few data points in the satwnd dump file from Nc005080 (~800 points). I understand now why we do not see AVHHR winds from NOAA satellites assimilated. But since we are still using AVHRR radiance data from NOAA satellites. What went wrong with the AVHRR AVMs from NOAA satellites (just curious)?

BrettHoover-NOAA commented 4 months ago

Hi Emily,

I am going to go back and fill in missing AMV subtypes now that we have all of the basic types working (except for Leo/Geo), some of which will require running other test-dates. I have test YAMLs for MetOp AVHRR ready and will deploy in a separate PR to add them in along with necessary modifications to the existing AVHRR satwind code. There are more GEO subtypes that need to be added in as well.

Both MODIS and n15/18/19 AVHRR satwinds were recently discontinued and feeds are being shut down, which is why you may only see a few of them in current diag files.

Brett Hoover Physical Scientist III, Contractor with Lynker in support of NOAA NWS/NCEP/EMC | U.S. Department of Commerce Office: (920) 541-7650

On Fri, Apr 19, 2024 at 6:24 PM emilyhcliu @.***> wrote:

@BrettHoover-NOAA https://github.com/BrettHoover-NOAA I have two questions for AVHRR winds. For 2021080100, we also have AVHRR AMVs from MetOP-A and MetOp-B. I dumped out the satwnd bufr file and found that we have SAIDs: 206, 209, 233, 3, 4 (they are NOAA-15, NOAA-18, NOAA-19, MetOp-A, and MetOp-B, respectively) Do you plan to add AVHRR from MetOp satellites later?

Here is the convinfo valid for 2021080100

uv 244 3 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 4 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 5 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 206 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 207 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 209 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 223 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8

For current operational, we do not assimilate AVHRR from NOAA satellites anymore. But it is still registrated in the convinfo file. It is a bit odd. I see AVHRR winds from NOAA-15, 18, and 19 in the satwnd bufr dump file (NC005080). The MetOp-B and MetOp-C are in the same dump file under NC005081. Is there any reason we do not use AVHRR winds from NOAA satellites anymore? (we are still using AVHRR radiance data from NOAA satellites NOAA-19 and NOAA-18).

Current OPS (v16.3.14)

uv 244 3 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 4 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 5 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 206 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 207 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 209 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 223 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8

Thanks!!

Oh, I see. I checked the NC005080 (NOAA AVHRR winds) in the current operational run. There are very few data points in the satwnd dump file from Nc005080 (~800 points). I understand now why we do not see AVHHR winds from NOAA satellites assimilated. But since we are still using AVHRR radiance data from NOAA satellites. What went wrong with the AVHRR AVMs from NOAA satellites (just curious)?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/GDASApp/pull/997#issuecomment-2067388463, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXNDXO7SGJOTDUMC6WN4QYDY6GRRVAVCNFSM6AAAAABFJOTXACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRXGM4DQNBWGM . You are receiving this because you were mentioned.Message ID: @.***>

emilyhcliu commented 4 months ago

Hi Emily, I am going to go back and fill in missing AMV subtypes now that we have all of the basic types working (except for Leo/Geo), some of which will require running other test-dates. I have test YAMLs for MetOp AVHRR ready and will deploy in a separate PR to add them in along with necessary modifications to the existing AVHRR satwind code. There are more GEO subtypes that need to be added in as well. Both MODIS and n15/18/19 AVHRR satwinds were recently discontinued and feeds are being shut down, which is why you may only see a few of them in current diag files. Brett Hoover Physical Scientist III, Contractor with Lynker in support of NOAA NWS/NCEP/EMC | U.S. Department of Commerce Office: (920) 541-7650 … On Fri, Apr 19, 2024 at 6:24 PM emilyhcliu @.> wrote: @BrettHoover-NOAA https://github.com/BrettHoover-NOAA I have two questions for AVHRR winds. For 2021080100, we also have AVHRR AMVs from MetOP-A and MetOp-B. I dumped out the satwnd bufr file and found that we have SAIDs: 206, 209, 233, 3, 4 (they are NOAA-15, NOAA-18, NOAA-19, MetOp-A, and MetOp-B, respectively) Do you plan to add AVHRR from MetOp satellites later? Here is the convinfo valid for 2021080100 uv 244 3 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 4 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 5 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 206 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 207 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 209 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 223 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 For current operational, we do not assimilate AVHRR from NOAA satellites anymore. But it is still registrated in the convinfo file. It is a bit odd. I see AVHRR winds from NOAA-15, 18, and 19 in the satwnd bufr dump file (NC005080). The MetOp-B and MetOp-C are in the same dump file under NC005081. Is there any reason we do not use AVHRR winds from NOAA satellites anymore? (we are still using AVHRR radiance data from NOAA satellites NOAA-19 and NOAA-18). Current OPS (v16.3.14) uv 244 3 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 4 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 5 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 206 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 207 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 209 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 uv 244 223 1 3.0 0 0 0 2.5 20.0 1.4 2.5 0.005000 0 0. 0. 0 0. 0. 15 -8 Thanks!! Oh, I see. I checked the NC005080 (NOAA AVHRR winds) in the current operational run. There are very few data points in the satwnd dump file from Nc005080 (~800 points). I understand now why we do not see AVHHR winds from NOAA satellites assimilated. But since we are still using AVHRR radiance data from NOAA satellites. What went wrong with the AVHRR AVMs from NOAA satellites (just curious)? — Reply to this email directly, view it on GitHub <#997 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXNDXO7SGJOTDUMC6WN4QYDY6GRRVAVCNFSM6AAAAABFJOTXACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRXGM4DQNBWGM . You are receiving this because you were mentioned.Message ID: @.>

@BrettHoover-NOAA Thanks for your explanation. Your plan is good. We have some other data types that also need more recent dates for validation (e.g. GOES ABI radiances ..etc). The date we are using is 2024021900. If you do not have a preferred date in mind, please also use 2024021900.