NOAA-GFDL / MOM6-examples

Example configurations for MOM6 and SIS2
Other
87 stars 148 forks source link

Getting Segmentation Fault for ocean_only/benchmark #229

Closed vineetmoraybfab closed 6 years ago

vineetmoraybfab commented 6 years ago

Running ocean_only/benchmark on Xeon Broadwell processors is printing segmentation fault. Have built the application using intel compilers and optimization flags: "-O3 -xHost(which comes out to be -msse2"

NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 32768. &MPP_IO_NML HEADER_BUFFER_VAL = 16384, GLOBAL_FIELD_ON_ROOT_PE = T, IO_CLOCKS_ON = F, SHUFFLE = 0, DEFLATE_LEVEL = -1, CF_COMPLIANCE = F / NOTE from PE 0: MPP_IO_SET_STACK_SIZE: stack size set to 131072. NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 955296. NOTE from PE 0: ======== Model being driven by MOM_driver ======== NOTE from PE 0: callTree: o Program MOM_main, MOM_driver.F90 NOTE from PE 0: open_param_file: MOM_input has been opened successfully. NOTE from PE 0: open_param_file: MOM_override has been opened successfully. MOM domain decomposition whalo = 4, ehalo = 4, shalo = 4, nhalo = 4 X-AXIS = 52 52 51 50 51 52 52 Y-AXIS = 45 45 45 45 NOTE from PE 0: diag_manager_mod::diag_manager_init: prepend_date only supported when diag_manager_init is called with time_init present. MOM domain decomposition whalo = 4, ehalo = 4, shalo = 4, nhalo = 4 X-AXIS = 52 52 51 50 51 52 52 Y-AXIS = 45 45 45 45 MOM domain decomposition whalo = 4, ehalo = 4, shalo = 4, nhalo = 4 X-AXIS = 52 52 51 50 51 52 52 Y-AXIS = 45 45 45 45 MOM Date 1/01/01 00:00:00 0: En 5.483130E-01, MaxCFL 0.00000, Mass 7.909733697458E+19, Salt 35.00000000000, Temp 5.06383031873 Total Energy: 4402CF0C41B7B49B 4.3370095656201052E+19 Total Mass: 7.9097336974577992E+19, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00) Total Salt: 2.7684067941102295E+18, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00) Total Heat: 1.5721018104275223E+24, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00) Total age: 0.0000000000000000E+00 yr kg forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FA70D8155E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FA70D162C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F77097315E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F770907EC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F5A039065E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F5A03253C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FB384FA35E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FB3848F0C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FA0782D35E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FA077C20C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F3651D335E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F3651680C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F4F74FD55E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F4F74922C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FB0E27A85E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FB0E20F5C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FEAEDFA95E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FEAED8F6C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F6BF91A65E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F6BF8AF3C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FBBB54485E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FBBB4D95C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FA1C02475E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FA1BFB94C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F1C8499A5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F1C842E7C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FAB7B28C5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FAB7ABD9C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F2C8C5795E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F2C8BEC6C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F55E1D5B5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F55E16A8C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FB07F10D5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FB07EA5AC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F5CBC36A5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F5CBBCB7C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F3E865C55E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F3E85F12C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F95C77645E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F95C70B1C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FF2C10565E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FF2C09A3C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F79723765E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F7971CC3C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F0DA48E65E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F0DA4233C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F1FED0355E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F1FEC982C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FE3D2D5D5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FE3D26AAC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F374BFD75E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F374B924C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FB4C50035E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FB4C4950C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F727FA875E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F727F3D4C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown

This is with DEBUG=True in MOM_override.

NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 32768. &MPP_IO_NML HEADER_BUFFER_VAL = 16384, GLOBAL_FIELD_ON_ROOT_PE = T, IO_CLOCKS_ON = F, SHUFFLE = 0, DEFLATE_LEVEL = -1, CF_COMPLIANCE = F / NOTE from PE 0: MPP_IO_SET_STACK_SIZE: stack size set to 131072. NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 955296. NOTE from PE 0: ======== Model being driven by MOM_driver ======== NOTE from PE 0: callTree: o Program MOM_main, MOM_driver.F90 NOTE from PE 0: open_param_file: MOM_input has been opened successfully. NOTE from PE 0: open_param_file: MOM_override has been opened successfully. MOM domain decomposition whalo = 4, ehalo = 4, shalo = 4, nhalo = 4 X-AXIS = 52 52 51 50 51 52 52 Y-AXIS = 45 45 45 45 NOTE from PE 0: diag_manager_mod::diag_manager_init: prepend_date only supported when diag_manager_init is called with time_init present. MOM domain decomposition whalo = 4, ehalo = 4, shalo = 4, nhalo = 4 X-AXIS = 52 52 51 50 51 52 52 Y-AXIS = 45 45 45 45 MOM domain decomposition whalo = 4, ehalo = 4, shalo = 4, nhalo = 4 X-AXIS = 52 52 51 50 51 52 52 Y-AXIS = 45 45 45 45 h-point: mean= 2.5343143909337967E+04 min= 2.1039476278425322E+04 max= 2.7829208694415916E+04 x MOM_grid_init/set_grid_metrics: d[xy]T h-point: c= 1940400 sw= 1939680 se= 1939680 nw= 1942560 ne= 1942560 x MOM_grid_init/set_grid_metrics: d[xy]T h-point: mean= 2.5343143909337967E+04 min= 2.1039476278425318E+04 max= 2.7829208694415920E+04 y MOM_grid_init/set_grid_metrics: d[xy]T h-point: c= 1935720 sw= 1933920 se= 1933920 nw= 1938600 ne= 1938600 y MOM_grid_init/set_grid_metrics: d[xy]T u-point: mean= 2.5343143909337967E+04 min= 2.1039476278425322E+04 max= 2.7829208694415916E+04 u MOM_grid_init/set_grid_metrics: dxC[uv] u-point: c= 1940400 sw= 1939680 se= 1939680 nw= 1942560 ne= 1942560 u MOM_grid_init/set_grid_metrics: dxC[uv] v-point: mean= 2.5362046189593148E+04 min= 2.1009424704132529E+04 max= 2.7829274923049583E+04 v MOM_grid_init/set_grid_metrics: dxC[uv] v-point: c= 1938960 sw= 1939320 se= 1939320 nw= 1938960 ne= 1938960 v MOM_grid_init/set_grid_metrics: dxC[uv] u-point: mean= 2.5343143909337967E+04 min= 2.1039476278425318E+04 max= 2.7829208694415920E+04 u MOM_grid_init/set_grid_metrics: dxC[uv] u-point: c= 1935720 sw= 1933920 se= 1933920 nw= 1938600 ne= 1938600 u MOM_grid_init/set_grid_metrics: dxC[uv] v-point: mean= 2.5362046189593148E+04 min= 2.1009424704132529E+04 max= 2.7829274923049583E+04 v MOM_grid_init/set_grid_metrics: dxC[uv] v-point: c= 1937160 sw= 1937520 se= 1937520 nw= 1937160 ne= 1937160 v MOM_grid_init/set_grid_metrics: dxC[uv] B-point: mean= 2.5362046189593148E+04 min= 2.1009424704132529E+04 max= 2.7829274923049583E+04 x MOM_grid_init/set_grid_metrics: dxB[uv] B-point: c= 1937160 sw= 1937520 se= 1937520 nw= 1937160 ne= 1937160 x MOM_grid_init/set_grid_metrics: dxB[uv] B-point: mean= 2.5362046189593148E+04 min= 2.1009424704132529E+04 max= 2.7829274923049583E+04 y MOM_grid_init/set_grid_metrics: dxB[uv] B-point: c= 1938960 sw= 1939320 se= 1939320 nw= 1938960 ne= 1938960 y MOM_grid_init/set_grid_metrics: dxB[uv] h-point: mean= 3.9743232448686427E-05 min= 3.5933468715574921E-05 max= 4.7529700205771659E-05 x MOM_grid_init/set_grid_metrics: Id[xy]T h-point: c= 2051280 sw= 2052360 se= 2052360 nw= 2053080 ne= 2053080 x MOM_grid_init/set_grid_metrics: Id[xy]T h-point: mean= 3.9743232448686427E-05 min= 3.5933468715574921E-05 max= 4.7529700205771672E-05 y MOM_grid_init/set_grid_metrics: Id[xy]T h-point: c= 2056680 sw= 2058480 se= 2058480 nw= 2058120 ne= 2058120 y MOM_grid_init/set_grid_metrics: Id[xy]T u-point: mean= 3.9743232448686427E-05 min= 3.5933468715574921E-05 max= 4.7529700205771659E-05 u MOM_grid_init/set_grid_metrics: Id[xy]C[uv] u-point: c= 2051280 sw= 2052360 se= 2052360 nw= 2053080 ne= 2053080 u MOM_grid_init/set_grid_metrics: Id[xy]C[uv] v-point: mean= 3.9710926189510507E-05 min= 3.5933383200427922E-05 max= 4.7597685994862164E-05 v MOM_grid_init/set_grid_metrics: Id[xy]C[uv] v-point: c= 2051640 sw= 2049840 se= 2049840 nw= 2050200 ne= 2050200 v MOM_grid_init/set_grid_metrics: Id[xy]C[uv] u-point: mean= 3.9743232448686427E-05 min= 3.5933468715574921E-05 max= 4.7529700205771672E-05 u MOM_grid_init/set_grid_metrics: Id[xy]C[uv] u-point: c= 2056680 sw= 2058480 se= 2058480 nw= 2058120 ne= 2058120 u MOM_grid_init/set_grid_metrics: Id[xy]C[uv] v-point: mean= 3.9710926189510507E-05 min= 3.5933383200427922E-05 max= 4.7597685994862164E-05 v MOM_grid_init/set_grid_metrics: Id[xy]C[uv] v-point: c= 2056320 sw= 2054520 se= 2054520 nw= 2054880 ne= 2054880 v MOM_grid_init/set_grid_metrics: Id[xy]C[uv] B-point: mean= 3.9710926189510507E-05 min= 3.5933383200427922E-05 max= 4.7597685994862164E-05 x MOM_grid_init/set_grid_metrics: Id[xy]B[uv] B-point: c= 2056320 sw= 2054520 se= 2054520 nw= 2054880 ne= 2054880 x MOM_grid_init/set_grid_metrics: Id[xy]B[uv] B-point: mean= 3.9710926189510507E-05 min= 3.5933383200427922E-05 max= 4.7597685994862164E-05 y MOM_grid_init/set_grid_metrics: Id[xy]B[uv] B-point: c= 2051640 sw= 2049840 se= 2049840 nw= 2050200 ne= 2050200 y MOM_grid_init/set_grid_metrics: Id[xy]B[uv] h-point: mean= 6.4666873870187151E+08 min= 4.4265956207042176E+08 max= 7.7446485655735457E+08 MOM_grid_init/set_grid_metrics: areaT h-point: c= 1949400 sw= 1948680 se= 1948680 nw= 1950480 ne= 1950480 MOM_grid_init/set_grid_metrics: areaT B-point: mean= 6.4759218620522928E+08 min= 4.4139592639861417E+08 max= 7.7446854274267638E+08 MOM_grid_init/set_grid_metrics: areaBu B-point: c= 1931400 sw= 1931400 se= 1931400 nw= 1927800 ne= 1927800 MOM_grid_init/set_grid_metrics: areaBu h-point: mean= 1.5914447124361934E-09 min= 1.2912141739332015E-09 max= 2.2590724016505310E-09 MOM_grid_init/set_grid_metrics: IareaT h-point: c= 2076480 sw= 2076120 se= 2076120 nw= 2077560 ne= 2077560 MOM_grid_init/set_grid_metrics: IareaT B-point: mean= 1.5887472392869648E-09 min= 1.2912080282287958E-09 max= 2.2655397120654979E-09 MOM_grid_init/set_grid_metrics: IareaBu B-point: c= 2087640 sw= 2085480 se= 2085480 nw= 2089080 ne= 2089080 MOM_grid_init/set_grid_metrics: IareaBu h-point: mean= 4.5000000000000000E+01 min= 1.2500000000000000E-01 max= 8.9874999999999986E+01 MOM_grid_init/set_grid_metrics: geoLonT h-point: c= 800460 sw= 792720 se= 800280 nw= 792720 ne= 800280 MOM_grid_init/set_grid_metrics: geoLonT h-point: mean= -2.1448475528681353E+01 min= -4.0885463109809884E+01 max= -1.2499990084080956E-01 MOM_grid_init/set_grid_metrics: geoLatT h-point: c= 1850040 sw= 1843560 se= 1843560 nw= 1855080 ne= 1855080 MOM_grid_init/set_grid_metrics: geoLatT B-point: mean= 4.5125000000000000E+01 min= 0.0000000000000000E+00 max= 9.0000000000000000E+01 MOM_grid_init/set_grid_metrics: geoLonBu B-point: c= 721260 sw= 720180 se= 728640 nw= 720180 ne= 728640 MOM_grid_init/set_grid_metrics: geoLonBu B-point: mean= -2.1334599933236913E+01 min= -4.0979898069620141E+01 max= 0.0000000000000000E+00 MOM_grid_init/set_grid_metrics: geoLatBu B-point: c= 1850760 sw= 1861560 se= 1861560 nw= 1855800 ne= 1855800 MOM_grid_init/set_grid_metrics: geoLatBu u-point: mean= 4.5125000000000000E+01 min= 0.0000000000000000E+00 max= 9.0000000000000000E+01 u MOM_grid_init/set_grid_metrics: geoLonC[uv] u-point: c= 721260 sw= 720180 se= 728640 nw= 720180 ne= 728640 u MOM_grid_init/set_grid_metrics: geoLonC[uv] v-point: mean= 4.5000000000000000E+01 min= 1.2500000000000000E-01 max= 8.9874999999999986E+01 v MOM_grid_init/set_grid_metrics: geoLonC[uv] v-point: c= 800460 sw= 792720 se= 800280 nw= 792720 ne= 800280 v MOM_grid_init/set_grid_metrics: geoLonC[uv] u-point: mean= -2.1448475528681353E+01 min= -4.0885463109809884E+01 max= -1.2499990084080956E-01 u MOM_grid_init/set_grid_metrics: geoLatC[uv] u-point: c= 1850040 sw= 1843560 se= 1843560 nw= 1855080 ne= 1855080 u MOM_grid_init/set_grid_metrics: geoLatC[uv] v-point: mean= -2.1334599933236913E+01 min= -4.0979898069620141E+01 max= 0.0000000000000000E+00 v MOM_grid_init/set_grid_metrics: geoLatC[uv] v-point: c= 1850760 sw= 1861560 se= 1861560 nw= 1855800 ne= 1855800 v MOM_grid_init/set_grid_metrics: geoLatC[uv] h-point: mean= 1.8742828447618181E+03 min= 5.0000000000000000E-01 max= 5.4547183562354667E+03 MOM_initialize_fixed: depth h-point: c= 1505221 sw= 1503001 se= 1503001 nw= 1503834 ne= 1503834 MOM_initialize_fixed: depth h-point: mean= 7.1373456790123457E-01 min= 0.0000000000000000E+00 max= 1.0000000000000000E+00 MOM_initialize_fixed: mask2dT h-point: c= 462500 MOM_initialize_fixed: mask2dT u-point: mean= 7.0998456790123454E-01 min= 0.0000000000000000E+00 max= 1.0000000000000000E+00 u MOM_initialize_fixed: mask2dC[uv] u-point: c= 460070 u MOM_initialize_fixed: mask2dC[uv] v-point: mean= 7.0817901234567904E-01 min= 0.0000000000000000E+00 max= 1.0000000000000000E+00 v MOM_initialize_fixed: mask2dC[uv] v-point: c= 458900 v MOM_initialize_fixed: mask2dC[uv] B-point: mean= 7.0442901234567901E-01 min= 0.0000000000000000E+00 max= 1.0000000000000000E+00 MOM_initialize_fixed: mask2dBu B-point: c= 456470 MOM_initialize_fixed: mask2dBu B-point: mean= -5.1935394571824036E-05 min= -9.5278618879650738E-05 max= 0.0000000000000000E+00 MOM_initialize_fixed: f B-point: c= 2122200 MOM_initialize_fixed: f h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 MOM_initialize_fixed: dF_dx h-point: c= 0 MOM_initialize_fixed: dF_dx h-point: mean= 2.0823624514094935E-11 min= 1.7287465519113497E-11 max= 2.2866325108382357E-11 MOM_initialize_fixed: dF_dy h-point: c= 2192760 MOM_initialize_fixed: dF_dy MOM_initialize_coord: Rlay 2.2753266928533503E+04 592 MOM_initialize_coord: g_prime 9.8766479591356156E+00 700 MOM domain decomposition whalo = 4, ehalo = 4, shalo = 4, nhalo = 4 X-AXIS = 52 52 51 50 51 52 52 Y-AXIS = 45 45 45 45 u-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 u MOM_initialize_state [uv] u-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 u MOM_initialize_state [uv] v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v MOM_initialize_state [uv] v-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 v MOM_initialize_state [uv] h-point: mean= 8.5194674761900828E+01 min= 1.0000000000000000E-10 max= 2.7094558579452141E+03 MOM_initialize_state: h h-point: c= 30808523 sw= 31001201 se= 31001201 nw= 31012932 ne= 31012932 MOM_initialize_state: h h-point: mean= 1.2016580834073947E+01 min= 2.9999999999993889E+00 max= 2.5000000000000306E+01 MOM_initialize_state: T h-point: c= 41572800 sw= 41337000 se= 41337000 nw= 41349960 ne= 41349960 MOM_initialize_state: T h-point: mean= 3.5000000000000000E+01 min= 3.5000000000000000E+01 max= 3.5000000000000000E+01 MOM_initialize_state: S h-point: c= 5702400 sw= 5670720 se= 5670720 nw= 5670720 ne= 5670720 MOM_initialize_state: S MOM domain decomposition whalo = 4, ehalo = 4, shalo = 4, nhalo = 4 X-AXIS = 52 52 51 50 51 52 52 Y-AXIS = 45 45 45 45 u-point: mean= 3.2272025813692477E-02 min= 0.0000000000000000E+00 max= 9.9999999995788746E-01 u btcalc frhat[uv] u-point: c= 33295431 W= 33295431 u btcalc frhat[uv] v-point: mean= 3.2189955106621766E-02 min= 0.0000000000000000E+00 max= 9.9999999995589761E-01 v btcalc frhat[uv] v-point: c= 33222478 S= 33222478 v btcalc frhat[uv] h-point: mean= 8.5194674761900828E+01 min= 1.0000000000000000E-10 max= 2.7094558579452141E+03 btcalc h h-point: c= 30808523 sw= 31001201 se= 31001201 nw= 31012932 ne= 31012932 btcalc h h-point: mean= 1.3444135763876215E+01 min= 4.1003812634885524E+00 max= 2.3899545898174800E+01 Post extract_sfc SST h-point: c= 1791000 sw= 1717698 se= 1717698 nw= 1718404 ne= 1718404 Post extract_sfc SST h-point: mean= 3.5000000000000000E+01 min= 3.5000000000000000E+01 max= 3.5000000000000000E+01 Post extract_sfc SSS h-point: c= 259200 sw= 248512 se= 248512 nw= 248512 ne= 248512 Post extract_sfc SSS h-point: mean= 3.4118964005379208E-13 min= -1.8189894035458565E-12 max= 2.2737367544323206E-12 Post extract_sfc sea_lev h-point: c= 373250 sw= 360147 se= 360127 nw= 359733 ne= 359697 Post extract_sfc sea_lev h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 Post extract_sfc Hml h-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 Post extract_sfc Hml u-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 u Post extract_sfc SSU u-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 u Post extract_sfc SSU v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v Post extract_sfc SSU v-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 v Post extract_sfc SSU h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 Post extract_sfc frazil h-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 Post extract_sfc frazil

WARNING from PE 0: MPP_OPEN: File ./CPU_stats opened WRONLY already exists!

WARNING from PE 0: MPP_OPEN: File ./CPU_stats opened WRONLY already exists!

u-point: mean= 6.2918894440132295E-02 min= 8.7761861854524664E-04 max= 9.9998475124111127E-02 u After set forcing forces%tau[xy] u-point: c= 2214720 W= 2214720 u After set forcing forces%tau[xy] v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v After set forcing forces%tau[xy] v-point: c= 0 S= 0 v After set forcing forces%tau[xy] h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing forces%p_surf h-point: c= 0 After set forcing forces%p_surf h-point: mean= 8.7602643979869162E-03 min= 4.4912817983100498E-03 max= 1.0767569627147516E-02 After set forcing fluxes%ustar h-point: c= 2146680 After set forcing fluxes%ustar h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%sw h-point: c= 0 After set forcing fluxes%sw h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%lw h-point: c= 0 After set forcing fluxes%lw h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%latent h-point: c= 0 After set forcing fluxes%latent h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%latent_evap_diag h-point: c= 0 After set forcing fluxes%latent_evap_diag h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%latent_fprec_diag h-point: c= 0 After set forcing fluxes%latent_fprec_diag h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%latent_frunoff_diag h-point: c= 0 After set forcing fluxes%latent_frunoff_diag h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%sens h-point: c= 0 After set forcing fluxes%sens h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%evap h-point: c= 0 After set forcing fluxes%evap h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%lprec h-point: c= 0 After set forcing fluxes%lprec h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%fprec h-point: c= 0 After set forcing fluxes%fprec h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%vprec h-point: c= 0 After set forcing fluxes%vprec h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%seaice_melt h-point: c= 0 After set forcing fluxes%seaice_melt h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%p_surf h-point: c= 0 After set forcing fluxes%p_surf h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%lrunoff h-point: c= 0 After set forcing fluxes%lrunoff h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%frunoff h-point: c= 0 After set forcing fluxes%frunoff h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%heat_content_lrunoff h-point: c= 0 After set forcing fluxes%heat_content_lrunoff h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%heat_content_frunoff h-point: c= 0 After set forcing fluxes%heat_content_frunoff h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%heat_content_lprec h-point: c= 0 After set forcing fluxes%heat_content_lprec h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%heat_content_fprec h-point: c= 0 After set forcing fluxes%heat_content_fprec h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%heat_content_cond h-point: c= 0 After set forcing fluxes%heat_content_cond h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 After set forcing fluxes%heat_content_massout h-point: c= 0 After set forcing fluxes%heat_content_massout

WARNING from PE 0: MPP_OPEN: File ./ocean.stats opened WRONLY already exists!

WARNING from PE 0: MPP_OPEN: File ./ocean.stats opened WRONLY already exists!

MOM Date 1/01/01 00:00:00 0: En 5.483130E-01, MaxCFL 0.00000, Mass 7.909733697458E+19, Salt 35.00000000000, Temp 5.06383031873 Total Energy: 4402CF0C41B7B49B 4.3370095656201052E+19 Total Mass: 7.9097336974577992E+19, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00) Total Salt: 2.7684067941102295E+18, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00) Total Heat: 1.5721018104275223E+24, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00) Total age: 0.0000000000000000E+00 yr kg u-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 u Beginning of step_MOM [uv] u-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 u Beginning of step_MOM [uv] v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v Beginning of step_MOM [uv] v-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 v Beginning of step_MOM [uv] h-point: mean= 8.5194674761900828E+01 min= 1.0000000000000000E-10 max= 2.7094558579452141E+03 Beginning of step_MOM h h-point: c= 30808523 sw= 31001201 se= 31001201 nw= 31012932 ne= 31012932 Beginning of step_MOM h u-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 u Beginning of step_MOM [uv]h u-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 u Beginning of step_MOM [uv]h v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v Beginning of step_MOM [uv]h v-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 v Beginning of step_MOM [uv]h u-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 u Before steps [uv] u-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 u Before steps [uv] v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v Before steps [uv] v-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 v Before steps [uv] h-point: mean= 8.5194674761900828E+01 min= 1.0000000000000000E-10 max= 2.7094558579452141E+03 Before steps h h-point: c= 30808523 sw= 31001201 se= 31001201 nw= 31012932 ne= 31012932 Before steps h u-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 u Before steps [uv]h u-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 u Before steps [uv]h v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v Before steps [uv]h v-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 v Before steps [uv]h u-point: mean= 6.2918894440132295E-02 min= 8.7761861854524664E-04 max= 9.9998475124111127E-02 u Before steps forces%tau[xy] u-point: c= 2214720 W= 2214720 u Before steps forces%tau[xy] v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v Before steps forces%tau[xy] v-point: c= 0 S= 0 v Before steps forces%tau[xy] h-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 Before steps forces%p_surf h-point: c= 0 Before steps forces%psurf u-point: mean= 4.5377314481066648E+01 min= 2.4396444961540895E-02 max= 1.7150342230693926E+02 u Kh[uv] u-point: c= 35257804 u Kh[uv] v-point: mean= 4.4836833530353495E+01 min= 2.9500707440092422E-02 max= 1.5855182947774423E+02 v Kh[uv] v-point: c= 35256631 v Kh_[uv] u-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 u intslope[uv] u-point: c= 0 u intslope[uv] v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v intslope[uv] v-point: c= 0 v intslope[uv] h-point: mean= 8.5194674761900828E+01 min= 1.0000000000000000E-10 max= 2.7094558579452141E+03 thickness_diffuse_1 h h-point: c= 30808523 sw= 31001201 se= 31001201 nw= 31012932 ne= 31012932 thickness_diffuse_1 h h-point: mean= -3.1184810990541325E+02 min= -5.4547183562354667E+03 max= 3.5527136788005009E-13 thickness_diffuse_1 e h-point: c= 53478758 sw= 53531920 se= 53531920 nw= 53510754 ne= 53510754 thickness_diffuse_1 e h-point: mean= 1.2016580834073947E+01 min= 2.9999999999993889E+00 max= 2.5000000000000306E+01 thickness_diffuse T h-point: c= 41572800 sw= 41337000 se= 41337000 nw= 41349960 ne= 41349960 thickness_diffuse T h-point: mean= 3.5000000000000000E+01 min= 3.5000000000000000E+01 max= 3.5000000000000000E+01 thickness_diffuse S h-point: c= 5702400 sw= 5670720 se= 5670720 nw= 5670720 ne= 5670720 thickness_diffuse S forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FECB7E4E5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FECB779BC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FCBC1A9A5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FCBC13E7C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F82A06235E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F829FF70C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FB5C7EDD5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FB5C782AC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FA30BCE35E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FA30B630C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007EFCD22705E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007EFCD1BBDC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FF25FAEC5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FF25F439C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F9DBBE475E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F9DBB794C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F541A96D5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F541A2BAC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F9EF7A5E5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F9EF73ABC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FC64BCCD5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FC64B61AC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F1236A535E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F12363A0C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FCCACE6D5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FCCAC7BAC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F7F258BA5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F7F25207C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FAD4A6A55E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FAD49FF2C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F69469015E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F694624EC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F2F982C65E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F2F97C13C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F81B99D55E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F81B9322C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FCAEF3AA5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FCAEECF7C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FC64FAC85E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FC64F415C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007FE86BEBC5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007FE86B809C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F2F664B15E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F2F65DFEC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F12F3D345E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F12F3681C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F1E1C8205E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F1E1C16DC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F7DFE7DB5E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F7DFE128C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F380DF645E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F380D8B1C05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007F697BC025E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007F697B54FC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source MOM6 000000000117F4D4 Unknown Unknown Unknown libpthread-2.17.s 00007EFCA72105E0 Unknown Unknown Unknown MOM6 0000000000779E2B mom_thickness_dif 536 MOM_thickness_diffuse.F90 MOM6 0000000000765AF5 mom_thickness_dif 311 MOM_thickness_diffuse.F90 MOM6 0000000000B011EC mom_mp_step_mom_d 884 MOM.F90 MOM6 0000000000AF3B94 mom_mp_stepmom 672 MOM.F90 MOM6 00000000007C1EE0 MAIN 507 MOM_driver.F90 MOM6 000000000040C89E Unknown Unknown Unknown libc-2.17.so 00007EFCA6B5DC05 libc_start_main Unknown Unknown MOM6 000000000040C7A9 Unknown Unknown Unknown

After setting ulimit -s unlimited, I am ending up with:

v-point: c= 33222397 S= 33222397 v btcalc frhat[uv] u-point: mean= 8.5074641257582826E+01 min= 0.0000000000000000E+00 max= 2.7092621133018024E+03 u btcalc h[uv] u-point: c= 37011342 W= 37011342 u btcalc h[uv] v-point: mean= 8.5055807009653520E+01 min= -2.5315756865150547E+01 max= 2.8150631502617102E+03 v btcalc h[uv] v-point: c= 35406179 S= 35406179 v btcalc h[uv] h-point: mean= 8.5194674761900828E+01 min= 1.0000000000000000E-10 max= 2.7094552603333541E+03 btcalc h h-point: c= 45524222 sw= 45705458 se= 45705458 nw= 45697729 ne= 45697729 btcalc h

FATAL from PE 0: NaN detected: u Predictor [uv]

FATAL from PE 0: NaN detected: u Predictor [uv]

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

Hallberg-NOAA commented 6 years ago

Thank you for letting us know, @BoNeLeS . Three further details would be helpful for us. 1) What version of the MOM6 code are you using? (I.e., what git-hash?) 2) If you try using -O2 or -O0, are you getting the same behavior? (In the past few years we have sometimes found compiler problems with overly aggressive optimization settings; the various compiler development groups are not so well funded as they once were.) 3) How many processors were you using?

vineetmoraybfab commented 6 years ago

Thanks for the suggestion Hallberg. And yes, I got the problem fixed. It was (as you said) a compiler issue.

I accidently compiled the application using intel -xHost flag which compiled it for my cluster's login node. Afterwards when I compiled using -axCORE-AVX2 flag(my compute node proc), this issue was no more.

Working like a charm... Will try for KNL arch (hope for more performance) and benchmark the same. Thanks...

And sorry for marking an issue that was not actually a MOM issue. Please close this ticket.

Hallberg-NOAA commented 6 years ago

Thanks for letting us know, @BoNeLeS . Compiler bugs are rare (especially compared with user error), but they do happen!