search

ESCOMP / mizuRoute

Reach-based river routing model
http://escomp.github.io/mizuRoute/
GNU General Public License v3.0
44 stars 53 forks source link

Update of ESMF version causes mizuRoute to abort #452

Open ekluzek opened 10 months ago

ekluzek commented 10 months ago

And update of the ESMF version on both Cheyenne and izumi cause mizuRoute to abort in the initialization phase inside of mizuRoute.

The PET file shows the following..

.onnected using mesh
20240110 181907.429 INFO             PET0 (lnd_import_export:fldlist_realize)(lnd_import_export:realize_fields):clmImport Field = Flrr_volrmch is connected using mesh
20240110 181915.508 INFO             PET0 (lnd_comp_nuopc):(InitializeRealize)  done
20240110 181915.508 INFO             PET0 (rof_comp_nuopc):(InitializeRealize)  called
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_Glue.C:189 ESMCI_meshaddnodes() Value unrecognized or out of range  -  node ids must be >= 1
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_Glue.C:391 ESMCI_meshaddnodes() Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:574 ESMCI_mesh_create_from_ESMFMesh_ Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:670 ESMCI_mesh_create_from_ESMFMesh_ Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 /project/esmf/PROGS/build/90831/esmf-8.5.0/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:284 ESMCI_mesh_create_from_file() Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 ESMCI_MeshCap.C:2600 MeshCap::meshcreatefromfilenew() Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 ESMF_Mesh.F90:1969 ESMF_MeshCreateFromFile() Value unrecognized or out of range  - Internal subroutine call returned Error
20240110 181915.769 ERROR            PET0 rof_comp_nuopc.F90:591 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 ERROR            PET0 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:2898 Value unrecognized or out of range  - Phase 'IPDv01p3' Initialize for modelComp 4: ROF did not return ESMF_SUCCESS
20240110 181915.769 ERROR            PET0 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:1982 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 ERROR            PET0 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:2898 Value unrecognized or out of range  - Phase 'IPDv02p3' Initialize for modelComp 1: ESM0001 did not return ESMF_SUCCESS
20240110 181915.769 ERROR            PET0 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:1987 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 ERROR            PET0 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:486 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 ERROR            PET0 esmApp.F90:134 Value unrecognized or out of range  - Passing error in return code
20240110 181915.769 INFO             PET0 Finalizing ESMF
nmizukami commented 10 months ago

Hi Erik, yes, it is clearly from the esmf function, but do you know what exactly is the problem? is it node id problem (usually starting 0, but does not like 0 any more?)? Let me know if there is anything I should look into!

ekluzek commented 10 months ago

The ccs_config issue for this is that documents the ESMF updates that result in the problem is here:

https://github.com/ESMCI/ccs_config_cesm/issues/146

ekluzek commented 10 months ago

Here's some more information on one of the failing cases and what the mesh file looks like:

SMS_Ld1_D_P2x1.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default.20240110_230530_klwpfm]$ ncdump -v elementConn  /fs/cgd/csm/inputdata/share/meshes/5x5pt-amazon_navy_ESMFmesh_cd5_c20210107.nc
netcdf \5x5pt-amazon_navy_ESMFmesh_cd5_c20210107 {
dimensions:
    nodeCount = 36 ;
    elementCount = 25 ;
    maxNodePElement = 4 ;
    coordDim = 2 ;
    origGridRank = 2 ;
variables:
    int origGridDims(origGridRank) ;
    double nodeCoords(nodeCount, coordDim) ;
        nodeCoords:units = "degrees" ;
    int elementConn(elementCount, maxNodePElement) ;
        elementConn:long_name = "Node indices that define the element connectivity" ;
        elementConn:_FillValue = -1 ;
    int numElementConn(elementCount) ;
        numElementConn:long_name = "Number of nodes per element" ;
    double centerCoords(elementCount, coordDim) ;
        centerCoords:units = "degrees" ;
    double elementArea(elementCount) ;
    int elementMask(elementCount) ;
        elementMask:units = "unitless" ;

// global attributes:
        :gridType = "unstructured mesh" ;
        :version = "0.9" ;
        :inputFile = "domain.lnd.5x5pt-amazon_navy.SCRIP.070121.nc" ;
        :timeGenerated = "Thu Jan  7 13:58:21 2021" ;
data:

 elementConn =
  1, 2, 7, 8,
  2, 3, 9, 7,
  3, 4, 10, 9,
  4, 5, 11, 10,
  5, 6, 12, 11,
  8, 7, 13, 14,
  7, 9, 15, 13,
  9, 10, 16, 15,
  10, 11, 17, 16,
  11, 12, 18, 17,
  14, 13, 19, 20,
  13, 15, 21, 19,
  15, 16, 22, 21,
  16, 17, 23, 22,
  17, 18, 24, 23,
  20, 19, 25, 26,
  19, 21, 27, 25,
  21, 22, 28, 27,
  22, 23, 29, 28,
  23, 24, 30, 29,
  26, 25, 31, 32,
  25, 27, 33, 31,
  27, 28, 34, 33,
  28, 29, 35, 34,
  29, 30, 36, 35 ;
}
ekluzek commented 10 months ago

Here are the other mesh files...

./xmlquery -p MESH                      
Results in group run_domain
    ATM_DOMAIN_MESH: /fs/cgd/csm/inputdata/share/meshes/5x5pt-amazon_navy_ESMFmesh_cd5_c20210107.nc
    GLC_DOMAIN_MESH: UNSET
    ICE_DOMAIN_MESH: UNSET
    LND_DOMAIN_MESH: /fs/cgd/csm/inputdata/share/meshes/5x5pt-amazon_navy_ESMFmesh_cd5_c20210107.nc
    MASK_MESH: /fs/cgd/csm/inputdata/share/meshes/5x5pt-amazon_navy_ESMFmesh_cd5_c20210107.nc
    OCN_DOMAIN_MESH: UNSET
    ROF_DOMAIN_MESH: /fs/cgd/csm/inputdata/rof/mizuRoute/meshes/r05_amazon_c110308_ctrcrd_cdf5_ESMFmesh_c20200624.nc
    WAV_DOMAIN_MESH: UNSET
ekluzek commented 10 months ago

And here's the mesh that has just the small squares around the center point of the HRU's

ncdump -v hru_id,elementConn /fs/cgd/csm/inputdata/rof/mizuRoute/meshes/r05_amazon_c110308_ctrcrd_cdf5_ESMFmesh_c20200624.nc
netcdf r05_amazon_c110308_ctrcrd_cdf5_ESMFmesh_c20200624 {
dimensions:
    elementCount = 748 ;
    connectionCount = 2992 ;
    nodeCount = 2992 ;
    coordDim = 2 ;
variables:
    char ocgis_coordinate_system ;
        ocgis_coordinate_system:proj4 = "+proj=longlat +datum=WGS84 +no_defs " ;
        ocgis_coordinate_system:ocgis_role = "coordinate_system" ;
    double hru_id(elementCount) ;
    int elementConn(connectionCount) ;
        elementConn:long_name = "Node indices that define the element connectivity." ;
        elementConn:start_index = 0LL ;
    int numElementConn(elementCount) ;
        numElementConn:long_name = "Number of nodes per element." ;
    double nodeCoords(nodeCount, coordDim) ;
        nodeCoords:units = "degrees" ;
        nodeCoords:long_name = "Node coordinate values indexed by element connectivity." ;
        nodeCoords:coordinates = "x y" ;
    double centerCoords(elementCount, coordDim) ;
        centerCoords:_FillValue = NaN ;
        centerCoords:units = "degrees" ;

// global attributes:
        :gridType = "unstructured" ;
        :version = "0.9" ;
        :history = "Mon Jun 22 16:06:21 2020: ncks -A -v centerCoords out.reorder.nc output/ESMF_unstruct_mosart0.5_amazon_land_360.nc\n",
            "Fri May  8 16:35:35 2020: ncrename -v CELLVALUE,hru_id ESMF_unstruct_mosart0.5_amazon_land_360.nc" ;
        :NCO = "netCDF Operators version 4.7.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
        :history_of_appended_files = "Mon Jun 22 16:06:21 2020: Appended file out.reorder.nc had no \"history\" attribute\n",
            "" ;
data:

 hru_id = 123694, 122974, 122254, 121534, 120814, 120094, 119374, 118654, 
    117934, 117214, 116494, 115774, 115054, 114334, 113614, 112894, 112174, 
    111454, 110734, 110014, 109294, 108574, 123695, 122975, 122255, 121535, 
    120815, 120095, 119375, 118655, 117935, 117215, 116495, 115775, 115055, 
    114335, 113615, 112895, 112175, 111455, 110735, 110015, 109295, 108575, 
    123696, 122976, 122256, 121536, 120816, 120096, 119376, 118656, 117936, 
    117216, 116496, 115776, 115056, 114336, 113616, 112896, 112176, 111456, 
    110736, 110016, 109296, 108576, 123697, 122977, 122257, 121537, 120817, 
    120097, 119377, 118657, 117937, 117217, 116497, 115777, 115057, 114337, 
    113617, 112897, 112177, 111457, 110737, 110017, 109297, 108577, 123698, 
    122978, 122258, 121538, 120818, 120098, 119378, 118658, 117938, 117218, 
    116498, 115778, 115058, 114338, 113618, 112898, 112178, 111458, 110738, 
    110018, 109298, 108578, 123699, 122979, 122259, 121539, 120819, 120099, 
    119379, 118659, 117939, 117219, 116499, 115779, 115059, 114339, 113619, 
    112899, 112179, 111459, 110739, 110019, 109299, 108579, 123700, 122980, 
    122260, 121540, 120820, 120100, 119380, 118660, 117940, 117220, 116500, 
    115780, 115060, 114340, 113620, 112900, 112180, 111460, 110740, 110020, 
    109300, 108580, 123701, 122981, 122261, 121541, 120821, 120101, 119381, 
    118661, 117941, 117221, 116501, 115781, 115061, 114341, 113621, 112901, 
    112181, 111461, 110741, 110021, 109301, 108581, 123702, 122982, 122262, 
    121542, 120822, 120102, 119382, 118662, 117942, 117222, 116502, 115782, 
    115062, 114342, 113622, 112902, 112182, 111462, 110742, 110022, 109302, 
    108582, 123703, 122983, 122263, 121543, 120823, 120103, 119383, 118663, 
    117943, 117223, 116503, 115783, 115063, 114343, 113623, 112903, 112183, 
    111463, 110743, 110023, 109303, 108583, 123704, 122984, 122264, 121544, 
    120824, 120104, 119384, 118664, 117944, 117224, 116504, 115784, 115064, 
    114344, 113624, 112904, 112184, 111464, 110744, 110024, 109304, 108584, 
    123705, 122985, 122265, 121545, 120825, 120105, 119385, 118665, 117945, 
    117225, 116505, 115785, 115065, 114345, 113625, 112905, 112185, 111465, 
    110745, 110025, 109305, 108585, 123706, 122986, 122266, 121546, 120826, 
    120106, 119386, 118666, 117946, 117226, 116506, 115786, 115066, 114346, 
    113626, 112906, 112186, 111466, 110746, 110026, 109306, 108586, 123707, 
    122987, 122267, 121547, 120827, 120107, 119387, 118667, 117947, 117227, 
    116507, 115787, 115067, 114347, 113627, 112907, 112187, 111467, 110747, 
    110027, 109307, 108587, 123708, 122988, 122268, 121548, 120828, 120108, 
    119388, 118668, 117948, 117228, 116508, 115788, 115068, 114348, 113628, 
    112908, 112188, 111468, 110748, 110028, 109308, 108588, 123709, 122989, 
    122269, 121549, 120829, 120109, 119389, 118669, 117949, 117229, 116509, 
    115789, 115069, 114349, 113629, 112909, 112189, 111469, 110749, 110029, 
    109309, 108589, 123710, 122990, 122270, 121550, 120830, 120110, 119390, 
    118670, 117950, 117230, 116510, 115790, 115070, 114350, 113630, 112910, 
    112190, 111470, 110750, 110030, 109310, 108590, 123711, 122991, 122271, 
    121551, 120831, 120111, 119391, 118671, 117951, 117231, 116511, 115791, 
    115071, 114351, 113631, 112911, 112191, 111471, 110751, 110031, 109311, 
    108591, 123712, 122992, 122272, 121552, 120832, 120112, 119392, 118672, 
    117952, 117232, 116512, 115792, 115072, 114352, 113632, 112912, 112192, 
    111472, 110752, 110032, 109312, 108592, 123713, 122993, 122273, 121553, 
    120833, 120113, 119393, 118673, 117953, 117233, 116513, 115793, 115073, 
    114353, 113633, 112913, 112193, 111473, 110753, 110033, 109313, 108593, 
    123714, 122994, 122274, 121554, 120834, 120114, 119394, 118674, 117954, 
    117234, 116514, 115794, 115074, 114354, 113634, 112914, 112194, 111474, 
    110754, 110034, 109314, 108594, 123715, 122995, 122275, 121555, 120835, 
    120115, 119395, 118675, 117955, 117235, 116515, 115795, 115075, 114355, 
    113635, 112915, 112195, 111475, 110755, 110035, 109315, 108595, 123716, 
    122996, 122276, 121556, 120836, 120116, 119396, 118676, 117956, 117236, 
    116516, 115796, 115076, 114356, 113636, 112916, 112196, 111476, 110756, 
    110036, 109316, 108596, 123717, 122997, 122277, 121557, 120837, 120117, 
    119397, 118677, 117957, 117237, 116517, 115797, 115077, 114357, 113637, 
    112917, 112197, 111477, 110757, 110037, 109317, 108597, 123718, 122998, 
    122278, 121558, 120838, 120118, 119398, 118678, 117958, 117238, 116518, 
    115798, 115078, 114358, 113638, 112918, 112198, 111478, 110758, 110038, 
    109318, 108598, 123719, 122999, 122279, 121559, 120839, 120119, 119399, 
    118679, 117959, 117239, 116519, 115799, 115079, 114359, 113639, 112919, 
    112199, 111479, 110759, 110039, 109319, 108599, 123720, 123000, 122280, 
    121560, 120840, 120120, 119400, 118680, 117960, 117240, 116520, 115800, 
    115080, 114360, 113640, 112920, 112200, 111480, 110760, 110040, 109320, 
    108600, 123721, 123001, 122281, 121561, 120841, 120121, 119401, 118681, 
    117961, 117241, 116521, 115801, 115081, 114361, 113641, 112921, 112201, 
    111481, 110761, 110041, 109321, 108601, 123722, 123002, 122282, 121562, 
    120842, 120122, 119402, 118682, 117962, 117242, 116522, 115802, 115082, 
    114362, 113642, 112922, 112202, 111482, 110762, 110042, 109322, 108602, 
    123723, 123003, 122283, 121563, 120843, 120123, 119403, 118683, 117963, 
    117243, 116523, 115803, 115083, 114363, 113643, 112923, 112203, 111483, 
    110763, 110043, 109323, 108603, 123724, 123004, 122284, 121564, 120844, 
    120124, 119404, 118684, 117964, 117244, 116524, 115804, 115084, 114364, 
    113644, 112924, 112204, 111484, 110764, 110044, 109324, 108604, 123725, 
    123005, 122285, 121565, 120845, 120125, 119405, 118685, 117965, 117245, 
    116525, 115805, 115085, 114365, 113645, 112925, 112205, 111485, 110765, 
    110045, 109325, 108605, 123726, 123006, 122286, 121566, 120846, 120126, 
    119406, 118686, 117966, 117246, 116526, 115806, 115086, 114366, 113646, 
    112926, 112206, 111486, 110766, 110046, 109326, 108606, 123727, 123007, 
    122287, 121567, 120847, 120127, 119407, 118687, 117967, 117247, 116527, 
    115807, 115087, 114367, 113647, 112927, 112207, 111487, 110767, 110047, 
    109327, 108607 ;

 elementConn = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 
    18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 
    36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 
    54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 
    72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 
    90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 
    106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 
    120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 
    134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 
    148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 
    162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 
    176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 
    190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 
    204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 
    218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 
    232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 
    246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 
    260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 
    274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 
    288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 
    302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 
    316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 
    330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 
    344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 
    358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 
    372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 
    386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 
    400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 
    414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 
    428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 
    442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 
    456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 
    470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 
    484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 
    498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 
    512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 
    526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 
    540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 
    554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 
    568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 
    582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 
    596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 
    610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 
    624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 
    638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 
    652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 
    666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 
    680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 
    694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 
    708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 
    722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 
    736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 
    750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 
    764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 
    778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 
    792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 
    806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 
    820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 
    834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 
    848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 
    862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 
    876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 
    890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 
    904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 
    918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 
    932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 
    946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 
    960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 
    974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 
    988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 
    1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 
    1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 
    1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 
    1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049, 
    1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 
    1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 
    1074, 1075, 1076, 1077, 1078, 1079, 1080, 1081, 1082, 1083, 1084, 1085, 
    1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 
    1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 
    1110, 1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118, 1119, 1120, 1121, 
    1122, 1123, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 
    1134, 1135, 1136, 1137, 1138, 1139, 1140, 1141, 1142, 1143, 1144, 1145, 
    1146, 1147, 1148, 1149, 1150, 1151, 1152, 1153, 1154, 1155, 1156, 1157, 
    1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 
    1170, 1171, 1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 
    1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 
    1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1205, 
    1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 
    1218, 1219, 1220, 1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 
    1230, 1231, 1232, 1233, 1234, 1235, 1236, 1237, 1238, 1239, 1240, 1241, 
    1242, 1243, 1244, 1245, 1246, 1247, 1248, 1249, 1250, 1251, 1252, 1253, 
    1254, 1255, 1256, 1257, 1258, 1259, 1260, 1261, 1262, 1263, 1264, 1265, 
    1266, 1267, 1268, 1269, 1270, 1271, 1272, 1273, 1274, 1275, 1276, 1277, 
    1278, 1279, 1280, 1281, 1282, 1283, 1284, 1285, 1286, 1287, 1288, 1289, 
    1290, 1291, 1292, 1293, 1294, 1295, 1296, 1297, 1298, 1299, 1300, 1301, 
    1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 
    1314, 1315, 1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 
    1326, 1327, 1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 
    1338, 1339, 1340, 1341, 1342, 1343, 1344, 1345, 1346, 1347, 1348, 1349, 
    1350, 1351, 1352, 1353, 1354, 1355, 1356, 1357, 1358, 1359, 1360, 1361, 
    1362, 1363, 1364, 1365, 1366, 1367, 1368, 1369, 1370, 1371, 1372, 1373, 
    1374, 1375, 1376, 1377, 1378, 1379, 1380, 1381, 1382, 1383, 1384, 1385, 
    1386, 1387, 1388, 1389, 1390, 1391, 1392, 1393, 1394, 1395, 1396, 1397, 
    1398, 1399, 1400, 1401, 1402, 1403, 1404, 1405, 1406, 1407, 1408, 1409, 
    1410, 1411, 1412, 1413, 1414, 1415, 1416, 1417, 1418, 1419, 1420, 1421, 
    1422, 1423, 1424, 1425, 1426, 1427, 1428, 1429, 1430, 1431, 1432, 1433, 
    1434, 1435, 1436, 1437, 1438, 1439, 1440, 1441, 1442, 1443, 1444, 1445, 
    1446, 1447, 1448, 1449, 1450, 1451, 1452, 1453, 1454, 1455, 1456, 1457, 
    1458, 1459, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 
    1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1481, 
    1482, 1483, 1484, 1485, 1486, 1487, 1488, 1489, 1490, 1491, 1492, 1493, 
    1494, 1495, 1496, 1497, 1498, 1499, 1500, 1501, 1502, 1503, 1504, 1505, 
    1506, 1507, 1508, 1509, 1510, 1511, 1512, 1513, 1514, 1515, 1516, 1517, 
    1518, 1519, 1520, 1521, 1522, 1523, 1524, 1525, 1526, 1527, 1528, 1529, 
    1530, 1531, 1532, 1533, 1534, 1535, 1536, 1537, 1538, 1539, 1540, 1541, 
    1542, 1543, 1544, 1545, 1546, 1547, 1548, 1549, 1550, 1551, 1552, 1553, 
    1554, 1555, 1556, 1557, 1558, 1559, 1560, 1561, 1562, 1563, 1564, 1565, 
    1566, 1567, 1568, 1569, 1570, 1571, 1572, 1573, 1574, 1575, 1576, 1577, 
    1578, 1579, 1580, 1581, 1582, 1583, 1584, 1585, 1586, 1587, 1588, 1589, 
    1590, 1591, 1592, 1593, 1594, 1595, 1596, 1597, 1598, 1599, 1600, 1601, 
    1602, 1603, 1604, 1605, 1606, 1607, 1608, 1609, 1610, 1611, 1612, 1613, 
    1614, 1615, 1616, 1617, 1618, 1619, 1620, 1621, 1622, 1623, 1624, 1625, 
    1626, 1627, 1628, 1629, 1630, 1631, 1632, 1633, 1634, 1635, 1636, 1637, 
    1638, 1639, 1640, 1641, 1642, 1643, 1644, 1645, 1646, 1647, 1648, 1649, 
    1650, 1651, 1652, 1653, 1654, 1655, 1656, 1657, 1658, 1659, 1660, 1661, 
    1662, 1663, 1664, 1665, 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 
    1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 
    1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 
    1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 
    1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 
    1722, 1723, 1724, 1725, 1726, 1727, 1728, 1729, 1730, 1731, 1732, 1733, 
    1734, 1735, 1736, 1737, 1738, 1739, 1740, 1741, 1742, 1743, 1744, 1745, 
    1746, 1747, 1748, 1749, 1750, 1751, 1752, 1753, 1754, 1755, 1756, 1757, 
    1758, 1759, 1760, 1761, 1762, 1763, 1764, 1765, 1766, 1767, 1768, 1769, 
    1770, 1771, 1772, 1773, 1774, 1775, 1776, 1777, 1778, 1779, 1780, 1781, 
    1782, 1783, 1784, 1785, 1786, 1787, 1788, 1789, 1790, 1791, 1792, 1793, 
    1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 
    1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 
    1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 
    1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 
    1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 
    1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 
    1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 
    1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 
    1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 
    1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 
    1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 
    1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 
    1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 
    1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 
    1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 
    1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 
    1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 
    1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 
    2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 
    2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 
    2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 
    2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 
    2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 
    2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 
    2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 
    2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 
    2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 
    2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 
    2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 
    2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 
    2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 
    2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 
    2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 
    2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 
    2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 
    2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 
    2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 
    2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 
    2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 
    2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 
    2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 
    2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 
    2298, 2299, 2300, 2301, 2302, 2303, 2304, 2305, 2306, 2307, 2308, 2309, 
    2310, 2311, 2312, 2313, 2314, 2315, 2316, 2317, 2318, 2319, 2320, 2321, 
    2322, 2323, 2324, 2325, 2326, 2327, 2328, 2329, 2330, 2331, 2332, 2333, 
    2334, 2335, 2336, 2337, 2338, 2339, 2340, 2341, 2342, 2343, 2344, 2345, 
    2346, 2347, 2348, 2349, 2350, 2351, 2352, 2353, 2354, 2355, 2356, 2357, 
    2358, 2359, 2360, 2361, 2362, 2363, 2364, 2365, 2366, 2367, 2368, 2369, 
    2370, 2371, 2372, 2373, 2374, 2375, 2376, 2377, 2378, 2379, 2380, 2381, 
    2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 
    2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 
    2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 
    2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 
    2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2439, 2440, 2441, 
    2442, 2443, 2444, 2445, 2446, 2447, 2448, 2449, 2450, 2451, 2452, 2453, 
    2454, 2455, 2456, 2457, 2458, 2459, 2460, 2461, 2462, 2463, 2464, 2465, 
    2466, 2467, 2468, 2469, 2470, 2471, 2472, 2473, 2474, 2475, 2476, 2477, 
    2478, 2479, 2480, 2481, 2482, 2483, 2484, 2485, 2486, 2487, 2488, 2489, 
    2490, 2491, 2492, 2493, 2494, 2495, 2496, 2497, 2498, 2499, 2500, 2501, 
    2502, 2503, 2504, 2505, 2506, 2507, 2508, 2509, 2510, 2511, 2512, 2513, 
    2514, 2515, 2516, 2517, 2518, 2519, 2520, 2521, 2522, 2523, 2524, 2525, 
    2526, 2527, 2528, 2529, 2530, 2531, 2532, 2533, 2534, 2535, 2536, 2537, 
    2538, 2539, 2540, 2541, 2542, 2543, 2544, 2545, 2546, 2547, 2548, 2549, 
    2550, 2551, 2552, 2553, 2554, 2555, 2556, 2557, 2558, 2559, 2560, 2561, 
    2562, 2563, 2564, 2565, 2566, 2567, 2568, 2569, 2570, 2571, 2572, 2573, 
    2574, 2575, 2576, 2577, 2578, 2579, 2580, 2581, 2582, 2583, 2584, 2585, 
    2586, 2587, 2588, 2589, 2590, 2591, 2592, 2593, 2594, 2595, 2596, 2597, 
    2598, 2599, 2600, 2601, 2602, 2603, 2604, 2605, 2606, 2607, 2608, 2609, 
    2610, 2611, 2612, 2613, 2614, 2615, 2616, 2617, 2618, 2619, 2620, 2621, 
    2622, 2623, 2624, 2625, 2626, 2627, 2628, 2629, 2630, 2631, 2632, 2633, 
    2634, 2635, 2636, 2637, 2638, 2639, 2640, 2641, 2642, 2643, 2644, 2645, 
    2646, 2647, 2648, 2649, 2650, 2651, 2652, 2653, 2654, 2655, 2656, 2657, 
    2658, 2659, 2660, 2661, 2662, 2663, 2664, 2665, 2666, 2667, 2668, 2669, 
    2670, 2671, 2672, 2673, 2674, 2675, 2676, 2677, 2678, 2679, 2680, 2681, 
    2682, 2683, 2684, 2685, 2686, 2687, 2688, 2689, 2690, 2691, 2692, 2693, 
    2694, 2695, 2696, 2697, 2698, 2699, 2700, 2701, 2702, 2703, 2704, 2705, 
    2706, 2707, 2708, 2709, 2710, 2711, 2712, 2713, 2714, 2715, 2716, 2717, 
    2718, 2719, 2720, 2721, 2722, 2723, 2724, 2725, 2726, 2727, 2728, 2729, 
    2730, 2731, 2732, 2733, 2734, 2735, 2736, 2737, 2738, 2739, 2740, 2741, 
    2742, 2743, 2744, 2745, 2746, 2747, 2748, 2749, 2750, 2751, 2752, 2753, 
    2754, 2755, 2756, 2757, 2758, 2759, 2760, 2761, 2762, 2763, 2764, 2765, 
    2766, 2767, 2768, 2769, 2770, 2771, 2772, 2773, 2774, 2775, 2776, 2777, 
    2778, 2779, 2780, 2781, 2782, 2783, 2784, 2785, 2786, 2787, 2788, 2789, 
    2790, 2791, 2792, 2793, 2794, 2795, 2796, 2797, 2798, 2799, 2800, 2801, 
    2802, 2803, 2804, 2805, 2806, 2807, 2808, 2809, 2810, 2811, 2812, 2813, 
    2814, 2815, 2816, 2817, 2818, 2819, 2820, 2821, 2822, 2823, 2824, 2825, 
    2826, 2827, 2828, 2829, 2830, 2831, 2832, 2833, 2834, 2835, 2836, 2837, 
    2838, 2839, 2840, 2841, 2842, 2843, 2844, 2845, 2846, 2847, 2848, 2849, 
    2850, 2851, 2852, 2853, 2854, 2855, 2856, 2857, 2858, 2859, 2860, 2861, 
    2862, 2863, 2864, 2865, 2866, 2867, 2868, 2869, 2870, 2871, 2872, 2873, 
    2874, 2875, 2876, 2877, 2878, 2879, 2880, 2881, 2882, 2883, 2884, 2885, 
    2886, 2887, 2888, 2889, 2890, 2891, 2892, 2893, 2894, 2895, 2896, 2897, 
    2898, 2899, 2900, 2901, 2902, 2903, 2904, 2905, 2906, 2907, 2908, 2909, 
    2910, 2911, 2912, 2913, 2914, 2915, 2916, 2917, 2918, 2919, 2920, 2921, 
    2922, 2923, 2924, 2925, 2926, 2927, 2928, 2929, 2930, 2931, 2932, 2933, 
    2934, 2935, 2936, 2937, 2938, 2939, 2940, 2941, 2942, 2943, 2944, 2945, 
    2946, 2947, 2948, 2949, 2950, 2951, 2952, 2953, 2954, 2955, 2956, 2957, 
    2958, 2959, 2960, 2961, 2962, 2963, 2964, 2965, 2966, 2967, 2968, 2969, 
    2970, 2971, 2972, 2973, 2974, 2975, 2976, 2977, 2978, 2979, 2980, 2981, 
    2982, 2983, 2984, 2985, 2986, 2987, 2988, 2989, 2990, 2991 ;
}
nmizukami commented 10 months ago

Hi Erik (@ekluzek), hru_id is in double (I expected id to be int32). This might be problem? Not sure why it is in double (must be from python)..... but it is easy to change a type from double to int32? I wonder the other meshes also have ID in double precision.

UPDATE: I went through all the other mesh ID to check the type. all but HDMA amazon meth are in int64.

nmizukami commented 10 months ago

ah, looking at PET log, ESMF is complaining mesh node id (index?) not computing node.... that is strange (I never expect node index become negative)

ekluzek commented 10 months ago

Bob Oehmke from ESMF figured out the problem is that elementConn starts at 0 rather than the expected 1. This used to work, but because of potential issues more checking was added, and it now fails. ESMF is going to add an update where starting at 0 will also work. And also add clearer error checking so that it would easier to catch the problem.

@nmizukami has updates to the mesh files so they start at 1, so we'll try that out and verify this fixes the problem.

ekluzek commented 9 months ago

Trying out the 5x5_amazon case on Izumi starting elementConn at 1 works!

PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default CREATE_NEWCASE
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default XML
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default SETUP
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default SHAREDLIB_BUILD time=22
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default MODEL_BUILD time=133
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default SUBMIT
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default RUN time=148
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default MEMLEAK insuffiencient data for memleak test
PASS SMS_Ld1_D_Mmpi-serial.5x5_amazon_r05.I2000Clm50SpMizGs.izumi_nag.mizuroute-default SHORT_TERM_ARCHIVER
nmizukami commented 4 months ago

We need to move ESMF mesh with 1-index from naoki's directory to cesm data directory.

But also ESFM may allow us to use 0-index base MESH in next update??

ekluzek commented 3 months ago

ESMF told us that the fix for this went into ESMF 8.6.1. Which is great.

However, in terms of CESM this hasn't been brought in as of now. ccs_config has NOT updated to using that version on either Derecho nor Izumi. Derecho uses 8.6.0 and Izumi 8.5.0. There are a couple beta tags of 8.6.1 on Derecho, and a beta tag of 8.7.0 also so those are the only possibilities. On Izumi 8.5.0 is the latest version available, so trying it out would require installing a newer version of ESMF on at least Izumi, as well as using a beta version on Derecho.

ekluzek commented 3 months ago

FYI: Here's the PR that fixed it in ESMF https://github.com/esmf-org/esmf/issues/147 and the first beta tag with it was 8.6.1.b02 and that is available on Derecho. It does look like there is a 8.6.1 library on Derecho, but I'm not sure what it's compatible with.

ekluzek commented 2 days ago

The latest CESM is now using ESMF8.6.1 -- which should probably work correctly without updated datasets. We need to confirm that, and if so we could just do that rather than updating datasets.