RDA ECMWF hybrid vertical coords

[JohnLCaron]

ds626.0 ECMWF Atmospheric Reanalysis of the 20th Century

has array of 184 hybrid coords in the PDS. The level (byte 25-28) is an index goes from 1 - 91. Whats that mean?

Type of first fixed surface == 105 (table 4.5: Hybrid level)

4.0) Product definition template 4.0 - analysis or forecast at a horizontal level or in a horizontal layer at a point in time 
  1:                                                                                 PDS length == 770 
  5:                                                                                    Section == 4 
  6:                                                Number of coordinates values after Template == 184 
  8:                                                         Product Definition Template Number == 0 
 10:                                                                         Parameter category == 2 
 11:                                                                           Parameter number == 13 
 12:                                                                 Type of generating process == 0 (table 4.3: Analysis) 
 13:                   Background generating process identifier (defined by originating centre) == 255 (table ProcessId: Table ProcessId code 255 not found) 
 14:         Analysis or forecast generating process identifier (defined by originating centre) == 142 (table ProcessId: Table ProcessId code 142 not found) 
 15:                                   Hours of observational data cut-off after reference time == 0 
 17:                                 Minutes of observational data cut-off after reference time == 0 
 18:                                                            Indicator of unit of time range == 1 (table 4.4: Hour) 
 19:                                                 Forecast time in units defined by octet 18 == 0 
 23:                                                                Type of first fixed surface == 105 (table 4.5: Hybrid level) 
 24:                                                        Scale factor of first fixed surface == 0 
 25:                                                        Scaled value of first fixed surface == 70 
 29:                                                               Type of second fixed surface == 255 (table 4.5: Missing) 
 30:                                                       Scale factor of second fixed surface == 255 
 31:                                                       Scaled value of second fixed surface == -9999 
Hybrid Coordinates (184) 
     0.00000    0.00001    0.49521    0.92340    4.90832   10.70681    7.81143   78.98640  152.33531  217.43112  367.77637 1226.19922 2170.85205 2822.59595 3605.48633 3489.23047 12148.05469 22140.53125 33157.32813 39602.73438 46821.24609 54838.70703 63675.98047 62503.81250 146662.00000 237645.37500 335469.00000 440114.18750 537909.31250 596963.37500 659310.06250 724884.00000 793579.43750 865245.06250 939708.12500 1016694.56250 381567.00000 1040697.00000 1721918.00000 2423687.00000 3144176.00000 3881638.00000 4630423.00000 5387495.00000 6144010.00000 6892222.00000 7624759.00000 8332862.00000 8699199.00000 9018000.00000 9313807.00000 9583088.00000 9822487.00000 10028807.00000 10198936.00000 10329802.00000 10418645.00000 10462289.00000 10458224.00000 10403334.00000 10294800.00000 10130103.00000 9906573.00000 9624997.00000 9288344.00000 8901425.00000 8470828.00000 7620164.00000 6629013.00000 5592289.00000 4522377.00000 3431595.00000 2334014.00000 1242340.00000 168195.00000 938901.06250 813301.87500 694118.75000 582427.62500 434038.87500 246309.62500 78899.87500 57115.34766 41524.99609 24524.38672 4116.10547 2694.17896  544.69482   88.83334    0.82195    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00000    0.00001    0.00001    0.00001    0.00002    0.00005    0.00009    0.00013    0.00016    0.00019    0.00022    0.00011    0.00044    0.00080    0.00120    0.00163    0.00203    0.00228    0.00255    0.00283    0.00312    0.00341    0.00371    0.00083    0.00322    0.00558    0.00791    0.01018    0.01239    0.01451    0.01654    0.01846    0.02026    0.02193    0.02347    0.02486    0.02610    0.02720    0.02814    0.02894    0.02959    0.03014    0.03057    0.03089    0.03110    0.03125 

currenly we just use for the z coordinate:

float hybrid(hybrid=91);
  :units = "sigma";
  :long_name = "Hybrid level";
  :positive = "down";
  :Grib_level_type = 105; // int
  :_CoordinateAxisType = "GeoZ";
  :_CoordinateZisPositive = "down";

 midpoints== 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 11.000000 12.000000 13.000000 14.000000 15.000000 16.000000 17.000000 18.000000 19.000000 20.000000 21.000000 22.000000 23.000000 24.000000 25.000000 26.000000 27.000000 28.000000 29.000000 30.000000 31.000000 32.000000 33.000000 34.000000 35.000000 36.000000 37.000000 38.000000 39.000000 40.000000 41.000000 42.000000 43.000000 44.000000 45.000000 46.000000 47.000000 48.000000 49.000000 50.000000 51.000000 52.000000 53.000000 54.000000 55.000000 56.000000 57.000000 58.000000 59.000000 60.000000 61.000000 62.000000 63.000000 64.000000 65.000000 66.000000 67.000000 68.000000 69.000000 70.000000 71.000000 72.000000 73.000000 74.000000 75.000000 76.000000 77.000000 78.000000 79.000000 80.000000 81.000000 82.000000 83.000000 84.000000 85.000000 86.000000 87.000000 88.000000 89.000000 90.000000 91.000000

[JohnLCaron]

not clear where to get the surface pressure or ptop, to create CF hybrid coord.

for the moment, at least get the hybrid coords into the z coord if possible.

perhaps convert them in the PDS?

[JohnLCaron]

http://rda.ucar.edu/datasets/ds626.0/docs/ERA-20C.TL159L91_mhld.txt

---------------------------------------------
ERA 20th Century Model Half Level Definitions
---------------------------------------------
Note: In ERA-20C, there are 91 model levels.
However, ECMWF provides  the hybrid 'a' and
'b' coefficients at model HALF levels which
bracket the model levels. Coefficients are
arranged from the top of the model to the
surface. The corresponding pressure is given
by:
           p = a + b*PS,
where PS is the surface pressure in Pa.
---------------------------------------------
Half Level          a (Pa)      b (Pa/Pa)
---------------------------------------------
  0                 0.000000    0.000000
  1                 2.000040    0.000000
  2                 3.980832    0.000000
  3                 7.387186    0.000000
  4                12.908319    0.000000
  5                21.413612    0.000000
  6                33.952858    0.000000
  7                51.746601    0.000000
  8                76.167656    0.000000
  9               108.715561    0.000000
 10               150.986023    0.000000
 11               204.637451    0.000000
 12               271.356506    0.000000
 13               352.824493    0.000000
 14               450.685791    0.000000
 15               566.519226    0.000000
 16               701.813354    0.000000
 17               857.945801    0.000000
 18              1036.166504    0.000000
 19              1237.585449    0.000000
 20              1463.163940    0.000000
 21              1713.709595    0.000000
 22              1989.874390    0.000000
 23              2292.155518    0.000000
 24              2620.898438    0.000000
 25              2976.302246    0.000000
 26              3358.425781    0.000000
 27              3767.196045    0.000000
 28              4202.416504    0.000000
 29              4663.776367    0.000000
 30              5150.859863    0.000000
 31              5663.156250    0.000000
 32              6199.839355    0.000000
 33              6759.727051    0.000000
 34              7341.469727    0.000000
 35              7942.926270    0.000014
 36              8564.624023    0.000055
 37              9208.305664    0.000131
 38              9873.560547    0.000279
 39             10558.881836    0.000548
 40             11262.484375    0.001000
 41             11982.662109    0.001701
 42             12713.897461    0.002765
 43             13453.225586    0.004267
 44             14192.009766    0.006322
 45             14922.685547    0.009035
 46             15638.053711    0.012508
 47             16329.560547    0.016860
 48             16990.623047    0.022189
 49             17613.281250    0.028610
 50             18191.029297    0.036227
 51             18716.968750    0.045146
 52             19184.544922    0.055474
 53             19587.513672    0.067316
 54             19919.796875    0.080777
 55             20175.394531    0.095964
 56             20348.916016    0.112979
 57             20434.158203    0.131935
 58             20426.218750    0.152934
 59             20319.011719    0.176091
 60             20107.031250    0.201520
 61             19785.357422    0.229315
 62             19348.775391    0.259554
 63             18798.822266    0.291993
 64             18141.296875    0.326329
 65             17385.595703    0.362203
 66             16544.585938    0.399205
 67             15633.566406    0.436906
 68             14665.645508    0.475016
 69             13653.219727    0.513280
 70             12608.383789    0.551458
 71             11543.166992    0.589317
 72             10471.310547    0.626559
 73              9405.222656    0.662934
 74              8356.252930    0.698224
 75              7335.164551    0.732224
 76              6353.920898    0.764679
 77              5422.802734    0.795385
 78              4550.215820    0.824185
 79              3743.464355    0.850950
 80              3010.146973    0.875518
 81              2356.202637    0.897767
 82              1784.854614    0.917651
 83              1297.656128    0.935157
 84               895.193542    0.950274
 85               576.314148    0.963007
 86               336.772369    0.973466
 87               162.043427    0.982238
 88                54.208336    0.989153
 89                 6.575628    0.994204
 90                 0.003160    0.997630
 91                 0.000000    1.000000

[JohnLCaron]

The inclusion of the hybrid coords in every record quadruples the size of the indices and the memory needed for creating the ncx index. already the number of records in ds626.0.an.ml.3hr is larger than any other collection (384M), and the indexes are consuming 300G. The internal memory required goes from 80? to 850? bytes. Fails even with 42G heap!

making it a timePartition = 1 year allows it to be indexed by the TDM. not sure of the heap actually needed, but worked with 42G.

could have special processing to not store the duplicate hybrid coords.

[JohnLCaron]

https://docs.google.com/spreadsheets/d/1Xxh0JCbbAwukjrpGxB_Kk0i2Jq_z3Mvx_kXT0z1Yi3s/edit#gid=768140819

Collection	nfiles	nrecords	idx(MB)	data(GB)	data/idx	idx bytes/rec	variables	runtimes
ds626.0.an.ml.3hr	39,960	384,338,159	296G	48TB	160	853	15	324336
ds626.0.an.pl.3hr	21312	192,006,912	17G	18TB	1210	79	16	324336
ds626.0.an.pt.3hr	11988	46,704,384	3675.105	4787	1302	79	9	324336
ds626.0.an.pv.3hr	777	22,70,352	177.658	232	1309	78	7	324336
ds626.0.an.sfc.3hr	9324	27,244,224	2132.152	2809	1317	78	84	324336
ds626.0.an.sfc.6hr	1554	22,70,352	193.177	231	1200	85	14	162168

[JohnLCaron]

The 1-year partitions ncx4 are ~56M. Thats because has to hold the maps. MRUTC

The overall partition is ~8.5M. 324K runtimes. MRUTP

[JohnLCaron]

WE could work hard and reduce gbx9 by 80-90%.

First order of work is probably to just get the vert coords correct.

[JohnLCaron]

also see TestGribMiscCoordinates.testHybrid1()