search

GEOS-ESM / GEOSana_GridComp

Repository containing code for data analysis for the GEOS Earth System Model data assimilation
Apache License 2.0
1 stars 3 forks source link

Feature/jjin3/atms n21 #153

Closed gmao-jjin3 closed 7 months ago

gmao-jjin3 commented 10 months ago

Add a new type of microwave radiance observation: ATMS NOAA21. Its observational errors and quality control procedures are set as same as those for ATMS NOAA20. However, bias correction coefficients are tuned by a standalone run with f5295_fp with the whole observational data set and ATMS NOAA21 data in July 2023. These coefficients are stored on discover: /discover/nobackup/projects/gmao/obsdev/jjin3/x49atms/run/atms_n21_bc/x49atms.ana_satbiaspc_rst.20230714_21z.txt /discover/nobackup/projects/gmao/obsdev/jjin3/x49atms/run/atms_n21_bc/x49atms.ana_satbias_rst.20230714_21z.txt

github-actions[bot] commented 10 months ago

Label error. Requires at least 1 of: 0 diff, 0 diff trivial, Non 0-diff, 0 diff structural, 0-diff trivial, Not 0-diff, 0-diff, automatic, 0-diff uncoupled. Found:

gmao-jjin3 commented 10 months ago

Here is the section for atms_n21 in the ana_satbias_rst.txt


 4016 atms_n21                 1   0.527909E+00   0.423426E+07   999
       -0.083285    0.000000   60.855732    0.679465   -1.316112    0.000000    0.000000    0.008566    2.197706   -0.490640
        0.713919   -0.184084
 4017 atms_n21                 2   0.306324E+00   0.424773E+07   999
       -0.337455    0.000000   95.098945   -9.764592   -0.401710    0.000000    0.000000    0.007052    0.614060   -0.483898
        3.551624   -0.101780
 4018 atms_n21                 3   0.195078E+01   0.508867E+07   999
        0.420995    0.000000   63.057548   -0.780999    0.663388    0.000000    0.000000    0.010177   -5.422934   -0.021823
        2.464657    0.163422
 4019 atms_n21                 4   0.288883E+01   0.590847E+07   999
       -0.560477    0.000000   36.774746   -0.025840    0.361321    0.000000    0.000000    0.020900   -3.828329   -0.164403
        0.806418    0.126418
 4020 atms_n21                 5   0.444513E+01   0.649529E+07   999
       -0.584029    0.000000   11.407946    0.019914    0.093594    0.000000    0.000000    0.028896   -0.509119   -0.298093
       -0.811619    0.072669
 4021 atms_n21                 6   0.600855E+01   0.651371E+07   999
       -0.642807    0.000000    1.327475    0.002331   -0.051909    0.000000    0.000000    0.010554    0.789631   -0.271532
       -1.077935    0.020635
 4022 atms_n21                 7   0.727069E+01   0.117768E+08   999
       -0.339782    0.000000   -0.076771   -0.000737    0.003776    0.000000    0.000000    0.059203    0.223558   -0.236102
       -0.492784   -0.003604
 4023 atms_n21                 8   0.587887E+01   0.135488E+08   999
       -0.858804    0.000000   -0.046720   -0.001295   -0.040177    0.000000    0.000000    0.030819    0.287606   -0.240382
       -0.398343    0.015942
 4024 atms_n21                 9   0.412383E+01   0.136356E+08   999
       -0.816208    0.000000    0.074045   -0.001508   -0.062132    0.000000    0.000000    0.226240    0.206961   -0.166284
       -0.377510   -0.005939
 4025 atms_n21                10  -0.762593E+00   0.146518E+08   999
       -0.325109    0.000000   -0.097559   -0.000785   -0.089907    0.000000    0.000000    0.000000    0.169328   -0.112330
       -0.440070   -0.025933
 4026 atms_n21                11  -0.253401E+01   0.147677E+08   999
       -0.232471    0.000000   -0.051334    0.000827   -0.060651    0.000000    0.000000    0.000000    0.189928   -0.110946
       -0.384243   -0.014130
 4027 atms_n21                12  -0.369294E+01   0.148689E+08   999
       -0.033618    0.000000   -0.065550    0.001811   -0.103979    0.000000    0.000000    0.000000    0.141437   -0.117104
       -0.394210   -0.023576
 4028 atms_n21                13  -0.509167E+01   0.144791E+08   999
       -0.310091    0.000000   -0.068374    0.001325   -0.050715    0.000000    0.000000    0.000000    0.186023   -0.119376
       -0.465063   -0.027405
 4029 atms_n21                14  -0.615262E+01   0.145678E+08   999
       -0.109848    0.000000   -0.060503    0.000127   -0.070700    0.000000    0.000000   -0.243403    0.281383   -0.106386
       -0.451813   -0.039452
 4030 atms_n21                15  -0.425791E+01   0.464924E+07   999
        0.259732    0.000000   -0.022856    0.008808   -0.015815    0.000000    0.000000    0.000000    0.418419   -0.143422
       -0.360799   -0.034853
 4031 atms_n21                16   0.916124E+00   0.340707E+07   999
       -0.368995    0.000000  129.404398   -1.847394   -0.615067    0.000000    0.000000   -0.005118    0.896994   -1.471870
        6.820442    0.075320
 4032 atms_n21                17   0.229602E+01   0.454674E+07   999
       -0.267456    0.000000    9.288085    0.291909   -0.599839    0.000000    0.000000   -0.009320   -1.755281    0.045752
       -0.081013   -0.122612
 4033 atms_n21                18   0.383794E+01   0.543492E+07   999
       -0.320290    0.000000   -4.470448    0.032425   -0.042183    0.000000    0.000000    0.006405   -0.518057    0.013196
       -0.567805   -0.103561
 4034 atms_n21                19   0.504737E+01   0.538058E+07   999
       -0.305811    0.000000   -4.834375    0.016287    0.009727    0.000000    0.000000    0.020741   -0.392509   -0.067185
       -0.387001   -0.064494
 4035 atms_n21                20   0.631568E+01   0.512232E+07   999
       -0.214198    0.000000   -4.981171    0.012855    0.029432    0.000000    0.000000    0.063710   -0.358298   -0.040858
       -0.307140   -0.066858
 4036 atms_n21                21   0.802823E+01   0.489174E+07   999
        0.149236    0.000000   -5.393173    0.004356    0.058318    0.000000    0.000000    0.272915   -0.288052   -0.158199
       -0.265380   -0.031612
 4037 atms_n21                22   0.955803E+01   0.466305E+07   999
        0.254069    0.000000   -4.776581   -0.003441    0.064548    0.000000    0.000000    1.963386   -0.297696   -0.149381
       -0.316829   -0.030338
gmao-yzhu commented 10 months ago

In satbiaspc file, I would suggest to replace all zeros of data counts to 150. This will avoid all treatments due to zero data counts at the first cycle.

gmao-jjin3 commented 10 months ago

@gmao-yzhu Thanks. Here is an updated part in the satbiaspc file:

 4016 atms_n21                 1  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4017 atms_n21                 2  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4018 atms_n21                 3  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4019 atms_n21                 4  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4020 atms_n21                 5  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4021 atms_n21                 6  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4022 atms_n21                 7  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4023 atms_n21                 8  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4024 atms_n21                 9  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4025 atms_n21                10  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4026 atms_n21                11  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4027 atms_n21                12  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4028 atms_n21                13  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4029 atms_n21                14  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4030 atms_n21                15  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4031 atms_n21                16  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4032 atms_n21                17  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4033 atms_n21                18  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4034 atms_n21                19  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4035 atms_n21                20  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4036 atms_n21                21  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
 4037 atms_n21                22  1.5000000E+02
      0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05  0.1000000E+05
      0.1000000E+05  0.1000000E+05
gmao-jjin3 commented 9 months ago

I think these new bias correction coefficient are good to be added in the file satbias.txt:

 4016 atms_n21                 1   0.527909E+00   0.423426E+07   999
       -0.401911    0.000000   65.740291    5.478985    1.913973    0.000000    0.000000    0.012554  -39.007273   -1.145161
      -12.121239    0.328237
 4017 atms_n21                 2   0.306324E+00   0.424773E+07   999
       -0.614049    0.000000  100.268756    8.872828    3.661302    0.000000    0.000000    0.012024  -65.642540   -0.532081
       -6.618570    0.144692
 4018 atms_n21                 3   0.195078E+01   0.508867E+07   999
       -0.309881    0.000000   56.410096   -2.618153   -0.751593    0.000000    0.000000    0.002166   16.262008    0.582675
       11.080571    0.162774
 4019 atms_n21                 4   0.288883E+01   0.590847E+07   999
       -1.053736    0.000000   31.583919   -0.370910   -0.006039    0.000000    0.000000    0.006162    1.724965    0.225114
        7.296563    0.095759
 4020 atms_n21                 5   0.444513E+01   0.649529E+07   999
       -0.723524    0.000000    8.785349    0.007973    0.042942    0.000000    0.000000    0.006361   -2.223134   -0.209017
        2.378256    0.070533
 4021 atms_n21                 6   0.600855E+01   0.651371E+07   999
       -0.460057    0.000000    0.652270    0.006941   -0.020462    0.000000    0.000000    0.002040   -0.153738   -0.299803
       -0.232636    0.033488
 4022 atms_n21                 7   0.727069E+01   0.117768E+08   999
       -0.293587    0.000000   -0.143188    0.000363    0.014935    0.000000    0.000000    0.002961    0.239578   -0.254102
       -0.514579    0.006562
 4023 atms_n21                 8   0.587887E+01   0.135488E+08   999
       -0.751496    0.000000   -0.050897   -0.001096   -0.018557    0.000000    0.000000    0.031474    0.237055   -0.255596
       -0.374622    0.021712
 4024 atms_n21                 9   0.412383E+01   0.136356E+08   999
       -0.733379    0.000000    0.006645   -0.001632   -0.039303    0.000000    0.000000    0.066040    0.140693   -0.160120
       -0.365375   -0.006293
 4025 atms_n21                10  -0.762593E+00   0.146518E+08   999
       -0.344603    0.000000   -0.110125    0.003842   -0.078547    0.000000    0.000000   -3.302785    0.170984   -0.107942
       -0.439898   -0.027132
 4026 atms_n21                11  -0.253401E+01   0.147677E+08   999
       -0.238614    0.000000   -0.075817    0.005620   -0.067846    0.000000    0.000000   -7.131829    0.207844   -0.106298
       -0.409061   -0.020390
 4027 atms_n21                12  -0.369294E+01   0.148689E+08   999
        0.079941    0.000000   -0.033169    0.008424   -0.160113    0.000000    0.000000   -2.242194    0.180370   -0.121939
       -0.428961   -0.008650
 4028 atms_n21                13  -0.509167E+01   0.144791E+08   999
       -0.214724    0.000000   -0.029015    0.002643   -0.080470    0.000000    0.000000   -0.008292    0.052751   -0.116422
       -0.355046   -0.006370
 4029 atms_n21                14  -0.615262E+01   0.145678E+08   999
       -0.005010    0.000000   -0.001791    0.001845   -0.072508    0.000000    0.000000    4.912558    0.129033   -0.151816
       -0.278692   -0.007698
 4030 atms_n21                15  -0.425791E+01   0.464924E+07   999
        0.259732    0.000000   -0.022856    0.008808   -0.015815    0.000000    0.000000    0.000000    0.418419   -0.143422
       -0.360799   -0.034853
 4031 atms_n21                16   0.916124E+00   0.340707E+07   999
       -1.595269    0.000000  125.118676    1.813353    1.165252    0.000000    0.000000    0.008299  -21.272231   -0.652487
       -1.892191    0.035252
 4032 atms_n21                17   0.229602E+01   0.454674E+07   999
       -0.417825    0.000000    5.666847    0.268898   -0.757315    0.000000    0.000000   -0.006503   -2.766785   -0.223207
        1.566672   -0.067534
 4033 atms_n21                18   0.383794E+01   0.543492E+07   999
       -0.252357    0.000000   -5.344865    0.022170   -0.070440    0.000000    0.000000   -0.008424    0.088117   -0.040774
       -0.879315   -0.110631
 4034 atms_n21                19   0.504737E+01   0.538058E+07   999
       -0.203302    0.000000   -5.474052    0.006255   -0.027680    0.000000    0.000000   -0.011520    1.090787    0.127933
       -1.228600   -0.157906
 4035 atms_n21                20   0.631568E+01   0.512232E+07   999
       -0.125725    0.000000   -5.495713    0.004383   -0.006818    0.000000    0.000000   -0.002753    1.158862    0.086219
       -1.077816   -0.121583
 4036 atms_n21                21   0.802823E+01   0.489174E+07   999
        0.091288    0.000000   -5.387889    0.001441    0.027487    0.000000    0.000000   -0.005124    0.472620    0.142716
       -0.582316   -0.136437
 4037 atms_n21                22   0.955803E+01   0.466305E+07   999
        0.158396    0.000000   -4.269220   -0.003503    0.047903    0.000000    0.000000   -0.056082    0.061733    0.062091
       -0.315268   -0.096178
gmao-jjin3 commented 9 months ago

Here are O-F of atms_n20 and atms_n21 in a standalone run using x0049 results and those new CRTM coefficient file for atms_n21:
omf_atms_n21_passive.x0049.pdf These figures show AMTS_n21 O-F values are generally comparable to or smaller than those departures for ATMS_n21. Mean O-F values are very close to zero after bias correction. @rtodling Can you copy those CRTM coefficient files into the default CRTM coefficient directory? They are in @gmao-wgu 's directory /discover/nobackup/projects/gmao/obsdev/wgu/crtm_REL-2.4.0_emc/fix_emc20221018_2.4.0/CRTM_Coeffs/Little_Endian/ atms_n21.SpcCoeff.bin atms_n21-srf.SpcCoeff.bin atms_n21-srf.TauCoeff.bin atms_n21.TauCoeff.bin

You may want to copy over "cris-fsr_n21.SpcCoeff.binzz" and "cris-fsr_n21.TauCoeff.bin" as well. But, I defer that to you and @gmao-wgu. I gave Ben Johnson our current CRTM version number, but I didn't get a reply after that. I assume he will add these files in CRTM 2.4.0 and newer versions.

gmao-jjin3 commented 9 months ago

Forecast score verified by self-analysis in Aug 2023. The forecast is made with new ATMS_NOAA21 observations.
scorecard_x49atms_b_x0049_20230801_20230831

gmao-jjin3 commented 9 months ago

Forecast score verified by ERA-5 reanalysis in Aug 2023. The forecast is made with new ATMS_NOAA21 observations.

scorecard_x49atms_bXE5_x0049XE5_20230801_20230831

gmao-jjin3 commented 7 months ago

@rtodling I just want to let you know that I did not add "atms_n21" in the ensemble in these two file in my PR:

GEOSadas/src/Applications/NCEP_Etc/NCEP_enkf/scripts/gmao/etc/obs1gsi_mean.rc GEOSadas/src/Applications/NCEP_Etc/NCEP_enkf/scripts/gmao/etc/obs1gsi_member.rc

rtodling commented 7 months ago

@rtodling I just want to let you know that I did not add "atms_n21" in the ensemble in these two file in my PR:

GEOSadas/src/Applications/NCEP_Etc/NCEP_enkf/scripts/gmao/etc/obs1gsi_mean.rc GEOSadas/src/Applications/NCEP_Etc/NCEP_enkf/scripts/gmao/etc/obs1gsi_member.rc

Yes - that's another place that needs attention. Thanks.