search

NOAA-EMC / JEDI-T2O

JEDI Transition to Operations activities.
GNU Lesser General Public License v2.1
5 stars 4 forks source link

[End-to-End Test Code Sprint] Bufr to Ioda converter for SEVIRI METEOSAT-8 and to METEOSAT-11 #99

Open azadeh-gh opened 8 months ago

azadeh-gh commented 8 months ago

Spinning Enhanced Visible and InfraRed Imager (SEVIRI) from METEOSAT-8 and to METEOSAT-11 GDASApp end-to-end testing.

azadeh-gh commented 6 months ago

GSI final obs error

GSI_ObsError_Seviri08_ch5_QC

JEDI final obs error

5_JEDI_ObsError_Seviri08_ch5_QC

azadeh-gh commented 6 months ago

@ADCollard The plots show that both JEDI and GSI final obs error were almost 2.5 k everywhere. The min and max values on the plots are: vmin=2.5000, vmax=2.5001 GSI final obs error values: 4577 2.500105 4579 2.500102 4580 2.500095 4581 2.500106 4582 2.500103 9121 2.500106 9122 2.500101 9123 2.500099 9127 2.500100 9131 2.500108 Name: GsiFinalObsError, Length: 3967, dtype: float32

ADCollard commented 6 months ago

@azadeh-gh Thanks! I just spotted the "+2.5" on the color bar that is what was confusing me. Sorry for that. This looks good.

azadeh-gh commented 5 months ago

SEVIRI M11 channel 5 JEDI vs GSI final obs error

The values of obs error are 2.5 every where in both 5_JEDI_ObsError_Seviri11_ch5_QC GSI_ObsError_Seviri11_ch5_QC

emilyhcliu commented 4 months ago

@azadeh-gh

I worked on the seviri radiance a bit while I am in Tawian. When I used the obs file generated from GSI diagnostic file and run end-to-end without thinning and with QC, good match was found between GSI and JEDI When I used the obs file generated from BUFR and run end-to-end without thinning and with QC, good match was not found between GSU and JEDI. There were too much more data filtered out in in JEDI.

Three suggestions: 1. Something is wrong in QC in the end-to-end test. The problem may come from the MetaData, such as cloud information (cloud/clear amount) used by the QC procedure. It looks like you found the problem in the cloud/clear amount converted from BUFR. That's really good. Please test the updated IODA file (converted from BUFR) and run the end-to-end without thinning. The results should closely match GSI results.
2. Regarding observations (SECIRI radiances), the BUFR file contains 6 time levels. So, when you verify results, make sure you are validating the JEDI result with GSI for the same time level.
3. As for thinning, please see the attached flowchart. I moved the thinning to the fourth step (blue). It was in the second step before. If we place the thinning in the second step, more data will be screened out, we should execute thinning at the forth step. I tested thinning using the obs file generated from GSI diagnostic file. The results match well with GSI. I did not save the results. I will re-run the case and post the plots here.

SEVIRI-QC-Flowchart pptx

azadeh-gh commented 4 months ago

@emilyhcliu Thank you Emily.

  1. I regenerate the IODA un-thinned passed QC plots and add them here.
  2. Then update the Thinning part in the YAML and update the thinned plots
emilyhcliu commented 4 months ago

Updated IODA vs GSI No thinning passed QCs

JEDI_OmF_Seviri08_CH5_Un_thinned_NoQCs

GSI_OmF_Seviri08_ch5_Un_Thinned_passedQCs

The number of obs passed QC does not match! Are these two plots from the same time level?

ADCollard commented 4 months ago

The distribution of QC'd obs is much better than before, but the O-Bs don't match particularly well either. So I am aslo guessing there is a time level issue.

emilyhcliu commented 4 months ago

These are results using IODA obs file generated from GSI diagnostic file

No Thinning without QC (UFO vs. GSI) - Channel 5 Time Level 1 ufo_seviri_m08_omf_bc_channel5_time1 gsi_seviri_m08_omf_gsi_bc_channel5_time1

No Thinning with QC: UFO vs. GSI - Channel 5 Time Level 1 ufo_seviri_m08_omf_qc_channel5_time1 gsi_seviri_m08_omf_gsi_qc_channel5_time1

emilyhcliu commented 4 months ago

My color bar is too long. I should make it shorter (like the colorbar in your plots) so my globe will be bigger.
The results (after QC) should match this close! (15435 vs. 15350).

emilyhcliu commented 4 months ago

These are results using IODA obs file generated from GSI diagnostic file

Same as above, but for time level 3

No Thinning without QC (UFO vs. GSI) - Channel 5 Time Level 3 ufo_seviri_m08_omf_bc_channel5_time3 gsi_seviri_m08_omf_gsi_bc_channel5_time3

No Thinning with QC: UFO vs. GSI - Channel 5 Time Level 3 ufo_seviri_m08_omf_qc_channel5_time3 gsi_seviri_m08_omf_gsi_qc_channel5_time3

emilyhcliu commented 4 months ago

This exercise of using IODA obs file from GSI diagnostics is to check if the QC filters work as expected without thinning. Based on the results. We can be sure that the QC filters are correct! So, let's dig into the BUFR converter python code a bit.

emilyhcliu commented 4 months ago

Updated IODA vs GSI No thinning passed QCs

JEDI_OmF_Seviri08_CH5_Un_thinned_NoQCs GSI_OmF_Seviri08_ch5_Un_Thinned_passedQCs

The number of obs passed QC does not match! Are these two plots from the same time level?

Your observation numbers are too big. I think you are plotting all-time levels. You need to separate the data into 6 time levels in your plotting code.

emilyhcliu commented 4 months ago

Here is the channel statistics after QC (unthinned) between GSI and UFO (using IODA obs file generated from GSI)

UFO and GSI match well for all channels

UFO Statistics (number of observation passed QC for each channel)

  0: QC seviri_m08 brightnessTemperature_4 : 16161 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_5 : 93326 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_6 : 23357 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_7 : 17378 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_8 : 17384 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_9 : 17195 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_10: 17009 passed out of 187230 observations.
  0: QC seviri_m08 brightnessTemperature_11: 17339 passed out of 187230 observations.

UFO Statistics (number of observations passed QC for each channel) --- look at the 3rd column

 1578    4 seviri_m08        16152      0     -1.800  -1.5661124  -0.5673065   0.2080393   0.8670387   0.6556824
 1579    5 seviri_m08        92862   1446      2.500  -0.4410675   0.0869686   0.2364663   1.2157258   1.2126111
 1580    6 seviri_m08        23359      0      2.250  -0.0169656  -0.0179425   0.1018162   0.7874951   0.7872907
 1581    7 seviri_m08        17370      0     -1.250  -0.7939590  -0.0862284   0.1748905   0.6390446   0.6332003
 1582    8 seviri_m08        17378      0     -1.250  -1.0336799  -0.0956096   0.1803227   0.5686671   0.5605721
 1583    9 seviri_m08        17196      0     -1.250  -0.4881847  -0.0800431   0.2121771   0.6633558   0.6585089
 1584   10 seviri_m08        17000      0     -1.450  -0.5078334  -0.0538672   0.1964553   0.6989892   0.6969105
 1585   11 seviri_m08        17334      0     -1.250  -0.5422348  -0.0475899   0.1762843   0.5332766   0.5311488
ADCollard commented 4 months ago

@azadeh-gh It looks like there are still residual QC differences at the very bottom. Can you zoom in a report the QC flag there in the GSI?

azadeh-gh commented 4 months ago

@ADCollard The PreQC=55 on those locations.

emilyhcliu commented 4 months ago

@azadeh-gh amd @ADCollard I found the problem in the Python code for the converter. I will update results and the fix here.

ADCollard commented 4 months ago

@azadeh-gh amd @ADCollard I found the problem in the Python code for the converter. I will update results and the fix here.

Nice @emilyhcliu . Is it still the cloud amount?

emilyhcliu commented 4 months ago

I checked the Python converter code. The fix for the cloud/clear amount is not correct. The cloud/clear amount should be one dimension (Location), not two dimensions (Location, Channel).

Here is the number of observations passed QC (unthinned) for m08 and m11 using the IODA obs file converted from BUFR (Python converter)

  0: QC seviri_m08 brightnessTemperature_4 : 16154 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_5 : 93326 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_6 : 23356 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_7 : 17373 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_8 : 17383 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_9 : 17195 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_10: 17011 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_11: 17339 passed out of 187225 observations.
---
  0: QC seviri_m11 brightnessTemperature_4 : 22476 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_5 : 102727 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_6 : 27377 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_7 : 23761 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_8 : 23798 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_9 : 23733 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_10: 23500 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_11: 23835 passed out of 197534 observations.

Here is the number of observations passed QC (3rd column; unthinned) for m08 and m11 from GSI:

 1578    4 seviri_m08        16152      0     -1.800  -1.5661124  -0.5673065   0.2080393   0.8670387   0.6556824
 1579    5 seviri_m08        92862   1446      2.500  -0.4410675   0.0869686   0.2364663   1.2157258   1.2126111
 1580    6 seviri_m08        23359      0      2.250  -0.0169656  -0.0179425   0.1018162   0.7874951   0.7872907
 1581    7 seviri_m08        17370      0     -1.250  -0.7939590  -0.0862284   0.1748905   0.6390446   0.6332003
 1582    8 seviri_m08        17378      0     -1.250  -1.0336799  -0.0956096   0.1803227   0.5686671   0.5605721
 1583    9 seviri_m08        17196      0     -1.250  -0.4881847  -0.0800431   0.2121771   0.6633558   0.6585089
 1584   10 seviri_m08        17000      0     -1.450  -0.5078334  -0.0538672   0.1964553   0.6989892   0.6969105
 1585   11 seviri_m08        17334      0     -1.250  -0.5422348  -0.0475899   0.1762843   0.5332766   0.5311488
---
 1602    4 seviri_m11        22486      0     -0.750  -2.3250160  -0.6533695   0.7378770   0.8289438   0.5101530
 1603    5 seviri_m11       102409   1911      2.500   0.2296573   0.0834692   0.2323322   1.2050544   1.2021602
 1604    6 seviri_m11        27390      0      2.250  -0.2998838  -0.0668661   0.1035764   0.7913032   0.7884730
 1605    7 seviri_m11        23766      0     -1.250  -0.6942221  -0.0758474   0.1457398   0.5892856   0.5843840
 1606    8 seviri_m11        23796      0     -1.250  -0.8387110  -0.1000189   0.1442385   0.5092437   0.4993249
 1607    9 seviri_m11        23741      0     -0.750  -0.4430402  -0.0774574   0.3777125   0.6373439   0.6326196
 1608   10 seviri_m11        23501      0     -0.800  -0.9435016  -0.0979794   0.4626198   0.6887142   0.6817091
 1609   11 seviri_m11        23832      0     -1.250  -0.9055599  -0.0906417   0.1152137   0.4306133   0.4209654

The numbers of observations passed QC (unthinned) match well between GSI and JEDI for all channels (active and passive)

I will post the plots for channel 5 here later (before and after QC)

azadeh-gh commented 4 months ago

@emilyhcliu but I checked the cloud amount values in bufr file and the cloud amount changes in each channel. The values of cloud amount in the GSI file are not correct.

azadeh-gh commented 4 months ago

Here is the example of an observation that rejected with preQC=55 in GSI but passed in JEDI: The value of ObsError is too big in GSI but 2.5 in JEDI image

emilyhcliu commented 4 months ago

Here is the updated Python code on ORION: /work2/noaa/da/eliu/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_sevcsr.py

I will leave the test with thinning to you. Please remember to move the thinning from the second step to the forth step as indicated in the updated flowchart

SEVIRI-QC-Flowchart pptx

emilyhcliu commented 4 months ago

@azadeh-gh amd @ADCollard I found the problem in the Python code for the converter. I will update results and the fix here.

Nice @emilyhcliu . Is it still the cloud amount?

The dimension of the cloud/clear amount is not correct.

azadeh-gh commented 4 months ago

@emilyhcliu in the python script you changed this line: q.add("cloudFree", "/RPSEQ7/NCLDMNT") to this: q.add("cloudFree", "/RPSEQ7{5}/NCLDMNT") Do we need to read only channel 5 cloudAmount from the bufr file?

emilyhcliu commented 4 months ago

@azadeh-gh It looks like there are still residual QC differences at the very bottom. Can you zoom in a report the QC flag there in the GSI?

@azadeh-gh You have more data QCed out over sea-ice area. I think your O-Fs near the south pole must be very different from GSI. Please plot the O-F without QC (unthinned). If you see large differences between GSI and JEDI. The difference must be coming from HofX. So, compare the HofX between GSI and JEDI.

@emilyhcliu in the python script you changed this line: q.add("cloudFree", "/RPSEQ7/NCLDMNT") to this: q.add("cloudFree", "/RPSEQ7{5}/NCLDMNT") Do we need to read only channel 5 cloudamount from bufr file?

@azadeh-gh Yes, the cloudFree (clear fraction) should be read from channel 5.

emilyhcliu commented 4 months ago

@emilyhcliu but I checked the cloud amount values in bufr file and the cloud amount changes in each channel. The values of cloud amount in the GSI file are not correct.

@azadeh-gh In the IODA obs file from GSI (see below), the dimensions of cloud and clear amount are "Location" only/

       float cloudAmount(Location) ;
                cloudAmount:_FillValue = 9.96921e+36f ;
        float cloudFree(Location) ;
                cloudFree:_FillValue = 9.96921e+36f ;
  data:

   cloudAmount = 18, 40, 64, 56, 2, 0, 9, 3, 27, 43, 70, 0, 0, 0, 7, 31, 70,
      67, 75, 95, 0, 0, 0, 0, 12, 81, 53, 80, 77, 84, 80, 0, 2, 12, 15, 20,
      4, 15, 40, 39, 20, 30, 65, 0, 0, 0, 0, 0, 5, 0, 0, 6, 6, 0, 17, 43, 54,
      51, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 11, 5, 16, 18, 0, 0, 0, 0, 0, 0,
      0, 0, 0, 0, 4, 14, 8, 5, 8, 32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 4,
      6, 2, 18, 0, 0, 0, 0, 1, 0, 0, 0, 25, 54, 70, 49, 1, 2, 4, 9, 36, 53,

These are cloud amounts at each location without channel dependency.

emilyhcliu commented 4 months ago

@azadeh-gh Please try the updated python converter code in your end-to-end test for unthinned case and post your plots for channel 5 here.

I will post mine here for your reference.

emilyhcliu commented 4 months ago

Please note that there are 6 time levels. Please post seviri m08 channel 5 for time levels 1, 3, and 6.

azadeh-gh commented 4 months ago

@emilyhcliu Thank you, Emily. I will update the plots. I wasn't aware that we're only reading and writing channel 5 cloud amount data from the BUFR file. Upon checking, noticed variations in cloud values across different channels.

emilyhcliu commented 4 months ago

Here are the results of the end-to-end for SEVIRI M08 Channel 5 (time levels 1, 3, and 6) using IODA obs file converted from BUFR

Unthinned without QC (UFO vs. GSI) --- time level 1

ufo_seviri_m08_omf_bc_channel5_time1 gsi_seviri_m08_omf_gsi_bc_channel5_time1

Unthnned with QC (UFO vs. GSI) --- time level 1

ufo_seviri_m08_omf_qc_channel5_time1 gsi_seviri_m08_omf_gsi_qc_channel5_time1

Unthinned without QC (UFO vs. GSI) --- time level 3

ufo_seviri_m08_omf_bc_channel5_time3 gsi_seviri_m08_omf_gsi_bc_channel5_time3

Unthnned with QC (UFO vs. GSI) --- time level 3 ufo_seviri_m08_omf_qc_channel5_time3 gsi_seviri_m08_omf_gsi_qc_channel5_time3

Unthinned without QC (UFO vs. GSI) --- time level 6 ufo_seviri_m08_omf_bc_channel5_time6 gsi_seviri_m08_omf_gsi_bc_channel5_time6

Unthnned with QC (UFO vs. GSI) --- time level 6 ufo_seviri_m08_omf_qc_channel5_time6 gsi_seviri_m08_omf_gsi_qc_channel5_time6

emilyhcliu commented 4 months ago

@azadeh-gh I updated the ratio of the plots (see plots above) so the globe is bigger.
The GSI and JEDI results match well. I will stop here. Please let me know if you run into any trouble with the thinned case.

ADCollard commented 4 months ago

@azadeh-gh @emilyhcliu I think you are both right. The confusion arises because in the GSI only channel 5 is used to generate the "cloud free" field even though the BUFR defines a cloud fraction for each channel. We (@azadeh-gh and me) were focussed on replicating the BUFR messages correctly and had not had not taken the second step yet.

        if(clrsky) then       
          call ufbrep(lnbufr,datasev1,1,ncld,iret,'NCLDMNT')
          rclrsky=bmiss
!          datasev1(1,5) is high-peaking water vapor channel
!          for SEVIRI CSR, clear-sky percentage are different between the high-peaking WV channel and other channels
           if(datasev1(1,5)>= zero .and. datasev1(1,5) <= 100.0_r_kind ) then
              rclrsky=datasev1(1,5)
!             first QC filter out data with less clear sky fraction
              if ( rclrsky < r70 ) cycle read_loop
           end if

Your original code did not specify a channel:

    q.add('cloudFree', '*/RPSEQ7/NCLDMNT')

which is why it was picking up the wrong channel in the original version. Emily explicitly adds the 5:

    q.add("cloudFree", "*/RPSEQ7{5}/NCLDMNT")

But (and this is my caveat), should the converter be producing what the GSI eventually wants (just channel 5) or should it properly replicate what is in the BUFR (channel dependent)?

I would argue for the latter (i.e., @azadeh-gh 's latest version of the converter and to explicitly specify channel 5 in that UFO YAML).

Thoughts?

ADCollard commented 4 months ago

One more thing. Looking in ObsSpace.yaml, cloudFree does appear to have dimensions of Location only.

- Variable: [ "cloudFree" ]
    Attributes: { units: "1" }

If we want to retain the original channel information we should use cloudAmount

Variable: [ "cloudAmount" ]
    Dimensions: [ [ "Location" ],  [ "Location", "Layer" ], [ "Location", "Channel"] ]
    Attributes: { units: "1" }
azadeh-gh commented 4 months ago

@emilyhcliu @ADCollard I got my previous results even after using the new IODA observations. This is because, despite using cloudAmount for all channels in the IODA obs files, The plots that I created/added yesterday were specifically for channel 5, so the results remained the same. The difference observed in the number of data points passing quality control is due to the updated YAML file. When I use Emily's updated YAML file, the results matched Emily's plots exactly.

azadeh-gh commented 4 months ago

@emilyhcliu @ADCollard Here are the results from the updated Python code and YAML file. I was able to replicate Emily's plots for unthinned observations across all time levels, and I've included some examples here. Subsequently, I added the thinning into the YAML file and generated corresponding plots.

Example of No thinning but passed QCs for time level 6, The observations that rejected in GSI but passed in JEDI had preQC= 5 (ifail_surface_qc=5, Reject due to not using over this surface in qc routine). a few obs rejected in JEDI but passed in GSI with EffectiveQC=19

GSI_OmF_Seviri08_ch5_Un_Thinned_Passed_QCsTimestep_level6

JEDI_OmF_Seviri08_CH5_Un_thinned_PassedQCsTimestep_level6

Thinned and passed QCs

GSI_OmF_Seviri08_ch5_Thinned_Passed_QCs JEDI_OmF_Seviri08_CH5_Thinned_PassedQCs

azadeh-gh commented 4 months ago

Channel 6

GSI_OmF_Seviri08_ch6_Thinned_Passed_QCs JEDI_OmF_Seviri08_CH6_Thinned_PassedQCs

emilyhcliu commented 4 months ago

e

What we have if OK. I use cloudFree

@emilyhcliu @ADCollard I got my previous results even after using the new IODA observations. This is because, despite using cloudAmount for all channels in the IODA obs files, The plots that I created/added yesterday were specifically for channel 5, so the results remained the same. The difference observed in the number of data points passing quality control is due to the updated YAML file. When I use Emily's updated YAML file, the results matched Emily's plots exactly.

@azadeh-gh, Two questions: 1, You got identical results for the runs using the original converter code (2D cloudAmount) and the updated code (1D cloudAmount) for channel 5?

  1. For the unthinned case, did you modify the YAML (filters) for these two runs?
emilyhcliu commented 4 months ago

@azadeh-gh @emilyhcliu I think you are both right. The confusion arises because in the GSI only channel 5 is used to generate the "cloud free" field even though the BUFR defines a cloud fraction for each channel. We (@azadeh-gh and me) were focussed on replicating the BUFR messages correctly and had not had not taken the second step yet.

        if(clrsky) then       
          call ufbrep(lnbufr,datasev1,1,ncld,iret,'NCLDMNT')
          rclrsky=bmiss
!          datasev1(1,5) is high-peaking water vapor channel
!          for SEVIRI CSR, clear-sky percentage are different between the high-peaking WV channel and other channels
           if(datasev1(1,5)>= zero .and. datasev1(1,5) <= 100.0_r_kind ) then
              rclrsky=datasev1(1,5)
!             first QC filter out data with less clear sky fraction
              if ( rclrsky < r70 ) cycle read_loop
           end if

Your original code did not specify a channel:

    q.add('cloudFree', '*/RPSEQ7/NCLDMNT')

which is why it was picking up the wrong channel in the original version. Emily explicitly adds the 5:

    q.add("cloudFree", "*/RPSEQ7{5}/NCLDMNT")

But (and this is my caveat), should the converter be producing what the GSI eventually wants (just channel 5) or should it properly replicate what is in the BUFR (channel dependent)?

I would argue for the latter (i.e., @azadeh-gh 's latest version of the converter and to explicitly specify channel 5 in that UFO YAML).

Thoughts?

Both ways are OK with me.

azadeh-gh commented 4 months ago

e

What we have if OK. I use cloudFree

@emilyhcliu @ADCollard I got my previous results even after using the new IODA observations. This is because, despite using cloudAmount for all channels in the IODA obs files, The plots that I created/added yesterday were specifically for channel 5, so the results remained the same. The difference observed in the number of data points passing quality control is due to the updated YAML file. When I use Emily's updated YAML file, the results matched Emily's plots exactly.

@azadeh-gh, Two questions: 1, You got identical results for the runs using the original converter code (2D cloudAmount) and the updated code (1D cloudAmount) for channel 5? 2. For the unthinned case, did you modify the YAML (filters) for these two runs?

  1. Yes, but only for channel 5, because the values of cloudAmount were same.
  2. Yes, after using your modified YAML and Python script got the identical results. Then added Thinning to the forth step of the updated YAML and plotted above plots.
emilyhcliu commented 4 months ago

e

What we have if OK. I use cloudFree

@emilyhcliu @ADCollard I got my previous results even after using the new IODA observations. This is because, despite using cloudAmount for all channels in the IODA obs files, The plots that I created/added yesterday were specifically for channel 5, so the results remained the same. The difference observed in the number of data points passing quality control is due to the updated YAML file. When I use Emily's updated YAML file, the results matched Emily's plots exactly.

@azadeh-gh, Two questions: 1, You got identical results for the runs using the original converter code (2D cloudAmount) and the updated code (1D cloudAmount) for channel 5? 2. For the unthinned case, did you modify the YAML (filters) for these two runs?

  1. Yes, but only for channel 5, because the values of cloudAmount were same.
  2. Yes, after using your modified YAML and Python script got the identical results. Then added Thinning to the forth step of the updated YAML and plotted above plots.

I asked because the two versions of the converter create cloud amount in different dimensions. One of them has an additional channel dimension, so the YAML where the variable is used should be modified accordingly.
I am a bit puzzled. The YAML should be modified accordingly to use your original converter (2D cloud amount) correctly in the filters.

azadeh-gh commented 4 months ago

@emilyhcliu @ADCollard Updated YAMLs and python converter in PR#766

azadeh-gh commented 4 months ago

seviri_m11 Channel 5

JEDI_OmF_Seviri11_CH5_Thinned_PassedQCs GSI_OmF_Seviri11_ch5_Thinned_Passed_QCs

seviri_m11 Channel 6

JEDI_OmF_Seviri11_CH6_Thinned_PassedQCs

GSI_OmF_Seviri11_ch6_Thinned_Passed_QCs

azadeh-gh commented 4 months ago

IODA vs GSI seviri_m08 Ch5 & Ch6

image

image

azadeh-gh commented 4 months ago

GSI_OmF_Seviri08_ch5_Thinned_Passed_QCs JEDI_OmF_Seviri08_CH5_Thinned_PassedQCs

emilyhcliu commented 4 months ago

Here are the channel statistics plots (#obs, bias/std for OMF and final obs error) for JEDI and GDAS runs (Unthinned)

O-F Bias/Standard Deviation JEDI GDAS
Final Obs Error JEDI GDAS
Number Obs Passed QC JEDI GDAS
azadeh-gh commented 4 months ago

Thinned passed Qcs

GSI_ChannelStates_seviri_m08_diag_2021080100_seviri_m08 JEDI_ChannelStates_seviri_m08_diag_2021080100_seviri_m08

GSI_ObsError_ChannelStates_seviri_m08_diag_2021080100_seviri_m08 JEDI_ObsError_ChannelStates_seviri_m08_diag_2021080100_seviri_m08

GSI_PassedQC_Countsseviri_m08_diag_2021080100_seviri_m08

JEDI_PassedQC_Countsseviri_m08_diag_2021080100_seviri_m08

JEDI 0: QC seviri_m08 brightnessTemperature_4: 546 passed out of 187225 observations. 0: QC seviri_m08 brightnessTemperature_5: 3986 passed out of 187225 observations. 0: QC seviri_m08 brightnessTemperature_6: 812 passed out of 187225 observations. 0: QC seviri_m08 brightnessTemperature_7: 629 passed out of 187225 observations. 0: QC seviri_m08 brightnessTemperature_8: 635 passed out of 187225 observations. 0: QC seviri_m08 brightnessTemperature_9: 616 passed out of 187225 observations. 0: QC seviri_m08 brightnessTemperature_10: 612 passed out of 187225 observations. 0: QC seviri_m08 brightnessTemperature_11: 630 passed out of 187225 observations. 0: QC seviri_m11 brightnessTemperature_4: 746 passed out of 197534 observations. 0: QC seviri_m11 brightnessTemperature_5: 4237 passed out of 197534 observations. 0: QC seviri_m11 brightnessTemperature_6: 922 passed out of 197534 observations. 0: QC seviri_m11 brightnessTemperature_7: 818 passed out of 197534 observations. 0: QC seviri_m11 brightnessTemperature_8: 822 passed out of 197534 observations. 0: QC seviri_m11 brightnessTemperature_9: 817 passed out of 197534 observations. 0: QC seviri_m11 brightnessTemperature_10: 808 passed out of 197534 observations. 0: QC seviri_m11 brightnessTemperature_11: 826 passed out of 197534 observations.

emilyhcliu commented 4 months ago

/Here are the channel statistics plots (#obs, bias/std for OMF and final obs error) for JEDI and GDAS runs (Thinned)

O-F Bias/Standard Deviation JEDI GDAS (original: ptime=0 & ithin_time=1)
Final Obs Error JEDI GDAS (original: ptime=0 & ithin_time=1)
Number Obs Passed QC JEDI GDAS (original: ptime=0 & ithin_time=1)
O-F Bias/Standard Deviation JEDI GDAS (ptime=0 & ithin_time=2)
Final Obs Error JEDI GDAS (ptime=0 & ithin_time=2)
Number Obs Passed QC JEDI GDAS (ptime=0 & ithin_time=2)

GSI (ptime=0 & ithin_time=1) --- original

 1578    4 seviri_m08          604      0     -1.800  -1.6664425  -0.6465958   0.2292820   0.9098325   0.6400851
 1579    5 seviri_m08         3967    110      2.500  -0.4593377   0.0501718   0.2999431   1.3692149   1.3682954
 1580    6 seviri_m08          892      0      2.250  -0.0446104  -0.0051772   0.1118269   0.8419722   0.8419563
 1581    7 seviri_m08          675      0     -1.250  -0.9303738  -0.1861693   0.2264663   0.7507246   0.7272746
 1582    8 seviri_m08          677      0     -1.250  -1.0797621  -0.1292236   0.2139787   0.6175371   0.6038654
 1583    9 seviri_m08          658      0     -1.250  -0.5937136  -0.1743215   0.2664909   0.7447730   0.7240848
 1584   10 seviri_m08          648      0     -1.450  -0.6084281  -0.1408040   0.2354207   0.7685143   0.7555054
 1585   11 seviri_m08          670      0     -1.250  -0.5726236  -0.0817513   0.2012528   0.5705738   0.5646868
--- 
 1602    4 seviri_m11          837      0     -0.750  -2.3486792  -0.5939860   0.7328441   0.8236663   0.5706197
 1603    5 seviri_m11         4253    158      2.500   0.3063720   0.1588176   0.2786130   1.3196343   1.3100426
 1604    6 seviri_m11         1059      0      2.250  -0.1432027   0.0857357   0.1072610   0.8230758   0.8185983
 1605    7 seviri_m11          912      0     -1.250  -0.7164994  -0.0783584   0.1766107   0.6654285   0.6607988
 1606    8 seviri_m11          916      0     -1.250  -0.8187622  -0.0682822   0.1578973   0.5309492   0.5265402
 1607    9 seviri_m11          913      0     -0.750  -0.4721010  -0.0954720   0.4703223   0.7222493   0.7159114
 1608   10 seviri_m11          903      0     -0.800  -0.9665710  -0.1124609   0.5617425   0.7783632   0.7701959
 1609   11 seviri_m11          921      0     -1.250  -0.8339662  -0.0163472   0.1259227   0.4512974   0.4510012

GSI (ptime=0 & ithin_time=2) --- closer to center of box and analysis time

 1578    4 seviri_m08          541      0     -1.800  -1.7938796  -0.7787300   0.2804260   1.0062840   0.6373280
 1579    5 seviri_m08         4006     73      2.500  -0.5140042  -0.0050128   0.2432654   1.2330830   1.2330728
 1580    6 seviri_m08          825      0      2.250  -0.0273681   0.0085220   0.0894444   0.7474635   0.7474149
 1581    7 seviri_m08          630      0     -1.250  -1.0174843  -0.2707135   0.2497690   0.7844096   0.7362150
 1582    8 seviri_m08          640      0     -1.250  -1.1344703  -0.1819824   0.2251281   0.6322797   0.6055246
 1583    9 seviri_m08          613      0     -1.250  -0.6675540  -0.2467316   0.2770614   0.7611647   0.7200661
 1584   10 seviri_m08          612      0     -1.450  -0.6967464  -0.2263676   0.2543705   0.7987981   0.7660523
 1585   11 seviri_m08          630      0     -1.250  -0.6514622  -0.1592031   0.2059898   0.5776025   0.5552288
 1602    4 seviri_m11          765      0     -0.750  -2.4577952  -0.6912033   0.8645461   0.8945727   0.5678894
 1603    5 seviri_m11         4305    102      2.500   0.2217094   0.0749274   0.2129663   1.1537406   1.1513050
 1604    6 seviri_m11          933      0      2.250  -0.2657382  -0.0308175   0.0848218   0.7280587   0.7274062
 1605    7 seviri_m11          826      0     -1.250  -0.8110140  -0.1725658   0.1776129   0.6605413   0.6376017
 1606    8 seviri_m11          829      0     -1.250  -0.8876646  -0.1335660   0.1586659   0.5307251   0.5136431
 1607    9 seviri_m11          827      0     -0.750  -0.5671903  -0.1910699   0.4541750   0.7173481   0.6914337
 1608   10 seviri_m11          821      0     -0.800  -1.0864129  -0.2276740   0.5498915   0.7728549   0.7385589
 1609   11 seviri_m11          834      0     -1.250  -0.9323592  -0.1171792   0.1180834   0.4373612   0.4213714

JEDI --- closer to center of box and analysis time

  0: QC seviri_m08 brightnessTemperature_4 : 549 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_5 : 3987 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_6 : 818 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_7 : 634 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_8 : 643 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_9 : 620 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_10: 617 passed out of 187225 observations.
  0: QC seviri_m08 brightnessTemperature_11: 636 passed out of 187225 observations.
---  
  0: QC seviri_m11 brightnessTemperature_4 : 771 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_5 : 4299 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_6 : 936 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_7 : 842 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_8 : 847 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_9 : 842 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_10: 834 passed out of 197534 observations.
  0: QC seviri_m11 brightnessTemperature_11: 852 passed out of 197534 observations.