Open JohnLCaron opened 8 years ago
not clear where to get the surface pressure or ptop, to create CF hybrid coord.
for the moment, at least get the hybrid coords into the z coord if possible.
perhaps convert them in the PDS?
http://rda.ucar.edu/datasets/ds626.0/docs/ERA-20C.TL159L91_mhld.txt
---------------------------------------------
ERA 20th Century Model Half Level Definitions
---------------------------------------------
Note: In ERA-20C, there are 91 model levels.
However, ECMWF provides the hybrid 'a' and
'b' coefficients at model HALF levels which
bracket the model levels. Coefficients are
arranged from the top of the model to the
surface. The corresponding pressure is given
by:
p = a + b*PS,
where PS is the surface pressure in Pa.
---------------------------------------------
Half Level a (Pa) b (Pa/Pa)
---------------------------------------------
0 0.000000 0.000000
1 2.000040 0.000000
2 3.980832 0.000000
3 7.387186 0.000000
4 12.908319 0.000000
5 21.413612 0.000000
6 33.952858 0.000000
7 51.746601 0.000000
8 76.167656 0.000000
9 108.715561 0.000000
10 150.986023 0.000000
11 204.637451 0.000000
12 271.356506 0.000000
13 352.824493 0.000000
14 450.685791 0.000000
15 566.519226 0.000000
16 701.813354 0.000000
17 857.945801 0.000000
18 1036.166504 0.000000
19 1237.585449 0.000000
20 1463.163940 0.000000
21 1713.709595 0.000000
22 1989.874390 0.000000
23 2292.155518 0.000000
24 2620.898438 0.000000
25 2976.302246 0.000000
26 3358.425781 0.000000
27 3767.196045 0.000000
28 4202.416504 0.000000
29 4663.776367 0.000000
30 5150.859863 0.000000
31 5663.156250 0.000000
32 6199.839355 0.000000
33 6759.727051 0.000000
34 7341.469727 0.000000
35 7942.926270 0.000014
36 8564.624023 0.000055
37 9208.305664 0.000131
38 9873.560547 0.000279
39 10558.881836 0.000548
40 11262.484375 0.001000
41 11982.662109 0.001701
42 12713.897461 0.002765
43 13453.225586 0.004267
44 14192.009766 0.006322
45 14922.685547 0.009035
46 15638.053711 0.012508
47 16329.560547 0.016860
48 16990.623047 0.022189
49 17613.281250 0.028610
50 18191.029297 0.036227
51 18716.968750 0.045146
52 19184.544922 0.055474
53 19587.513672 0.067316
54 19919.796875 0.080777
55 20175.394531 0.095964
56 20348.916016 0.112979
57 20434.158203 0.131935
58 20426.218750 0.152934
59 20319.011719 0.176091
60 20107.031250 0.201520
61 19785.357422 0.229315
62 19348.775391 0.259554
63 18798.822266 0.291993
64 18141.296875 0.326329
65 17385.595703 0.362203
66 16544.585938 0.399205
67 15633.566406 0.436906
68 14665.645508 0.475016
69 13653.219727 0.513280
70 12608.383789 0.551458
71 11543.166992 0.589317
72 10471.310547 0.626559
73 9405.222656 0.662934
74 8356.252930 0.698224
75 7335.164551 0.732224
76 6353.920898 0.764679
77 5422.802734 0.795385
78 4550.215820 0.824185
79 3743.464355 0.850950
80 3010.146973 0.875518
81 2356.202637 0.897767
82 1784.854614 0.917651
83 1297.656128 0.935157
84 895.193542 0.950274
85 576.314148 0.963007
86 336.772369 0.973466
87 162.043427 0.982238
88 54.208336 0.989153
89 6.575628 0.994204
90 0.003160 0.997630
91 0.000000 1.000000
The inclusion of the hybrid coords in every record quadruples the size of the indices and the memory needed for creating the ncx index. already the number of records in ds626.0.an.ml.3hr is larger than any other collection (384M), and the indexes are consuming 300G. The internal memory required goes from 80? to 850? bytes. Fails even with 42G heap!
making it a timePartition = 1 year allows it to be indexed by the TDM. not sure of the heap actually needed, but worked with 42G.
could have special processing to not store the duplicate hybrid coords.
Collection | nfiles | nrecords | idx(MB) | data(GB) | data/idx | idx bytes/rec | variables | runtimes |
---|---|---|---|---|---|---|---|---|
ds626.0.an.ml.3hr | 39,960 | 384,338,159 | 296G | 48TB | 160 | 853 | 15 | 324336 |
ds626.0.an.pl.3hr | 21312 | 192,006,912 | 17G | 18TB | 1210 | 79 | 16 | 324336 |
ds626.0.an.pt.3hr | 11988 | 46,704,384 | 3675.105 | 4787 | 1302 | 79 | 9 | 324336 |
ds626.0.an.pv.3hr | 777 | 22,70,352 | 177.658 | 232 | 1309 | 78 | 7 | 324336 |
ds626.0.an.sfc.3hr | 9324 | 27,244,224 | 2132.152 | 2809 | 1317 | 78 | 84 | 324336 |
ds626.0.an.sfc.6hr | 1554 | 22,70,352 | 193.177 | 231 | 1200 | 85 | 14 | 162168 |
The 1-year partitions ncx4 are ~56M. Thats because has to hold the maps. MRUTC
The overall partition is ~8.5M. 324K runtimes. MRUTP
WE could work hard and reduce gbx9 by 80-90%.
First order of work is probably to just get the vert coords correct.
also see TestGribMiscCoordinates.testHybrid1()
ds626.0 ECMWF Atmospheric Reanalysis of the 20th Century
has array of 184 hybrid coords in the PDS. The level (byte 25-28) is an index goes from 1 - 91. Whats that mean?
Type of first fixed surface == 105 (table 4.5: Hybrid level)
currenly we just use for the z coordinate: