search

s117 / anycore-riscv

The AnyCore toolset targetting the RISC-V ISA
Other
0 stars 0 forks source link

654.roms_s_ref failed (User load segfault @ 0x0000000000000005) #17

Closed s117 closed 4 years ago

s117 commented 4 years ago

The binary is compiled by the Linux Toolchain https://github.com/s117/riscv-gnu-toolchain/commit/d0bdaa9a282a32cc68e6203098dc1162021ceba7

Might be similar to https://github.com/s117/anycore-riscv/issues/15 (both might be caused by an unchecked malloc() call)

$ spike -m8192 pk -c sroms_base.riscv-m64 < ocean_benchmark3.in

Requesting target memory 0x200000000
******* Resetting core ********** 
****Initializing the processor system****
******* Resetting core ********** 
******* Resetting core ********** 
****Initialization complete****
 Process Information:

 Model Input Parameters:  ROMS/TOMS version 3.2  

 -----------------------------------------------------------------------------

 Benchmark Test, Idealized Southern Ocean, Large Grid
 Version adapted for spec, built without netcdf

 Operating system : 
 CPU/hardware     : 
 Compiler system  : 
 Compiler command : 
 Compiler flags   : 

 SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : 405M

 Local Root    : 
 Header Dir    : 
 Header file   : benchmark.h
 Analytical Dir: 

 Resolution, Grid 01: 2048x0512x040,  Parallel Threads:  0,  Tiling: 000x000

 Physical Parameters, Grid: 01
 =============================

        100  ntimes          Number of timesteps for 3-D equations.
     50.000  dt              Timestep size (s) for 3-D equations.
         20  ndtfast         Number of timesteps for 2-D equations between
                               each 3D timestep.
          1  ERstr           Starting ensemble/perturbation run number.
          1  ERend           Ending ensemble/perturbation run number.
          0  nrrec           Number of restart records to read from disk.
          T  LcycleRST       Switch to recycle time-records in restart file.
       1000  nRST            Number of timesteps between the writing of data
                               into restart fields.
          1  ninfo           Number of timesteps between print of information
                               to standard output.
          T  ldefout         Switch to create a new output NetCDF file(s).
       1000  nHIS            Number of timesteps between the writing fields
                               into history file.
 5.0000E+02  tnu2(01)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 01: temp
 5.0000E+02  tnu2(02)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 02: salt
 5.0000E+03  visc2           Horizontal, harmonic mixing coefficient (m2/s)
                               for momentum.
 1.0000E-05  Akt_bak(01)     Background vertical mixing coefficient (m2/s)
                               for tracer 01: temp
 1.0000E-05  Akt_bak(02)     Background vertical mixing coefficient (m2/s)
                               for tracer 02: salt
 1.0000E-04  Akv_bak         Background vertical mixing coefficient (m2/s)
                               for momentum.
 3.0000E-04  rdrg            Linear bottom drag coefficient (m/s).
 3.0000E-03  rdrg2           Quadratic bottom drag coefficient.
 2.0000E-02  Zob             Bottom roughness (m).
 1.0000E+01  blk_ZQ          Height (m) of surface air humidity measurement.
 1.0000E+01  blk_ZT          Height (m) of surface air temperature measurement.
 1.0000E+01  blk_ZW          Height (m) of surface winds measurement.
          1  lmd_Jwt         Jerlov water type.
          1  Vtransform      S-coordinate transformation equation.
          1  Vstretching     S-coordinate stretching function.
 5.0000E+00  theta_s         S-coordinate surface control parameter.
 4.0000E-01  theta_b         S-coordinate bottom  control parameter.
    200.000  Tcline          S-coordinate surface/bottom layer width (m) used
                               in vertical coordinate stretching.
   1025.000  rho0            Mean density (kg/m3) for Boussinesq approximation.
      0.000  dstart          Time-stamp assigned to model initialization (days).
       0.00  time_ref        Reference time for units attribute (yyyymmdd.dd)
 0.0000E+00  Tnudg(01)       Nudging/relaxation time scale (days)
                               for tracer 01: temp
 0.0000E+00  Tnudg(02)       Nudging/relaxation time scale (days)
                               for tracer 02: salt
 0.0000E+00  Znudg           Nudging/relaxation time scale (days)
                               for free-surface.
 0.0000E+00  M2nudg          Nudging/relaxation time scale (days)
                               for 2D momentum.
 0.0000E+00  M3nudg          Nudging/relaxation time scale (days)
                               for 3D momentum.
 0.0000E+00  obcfac          Factor between passive and active
                               open boundary conditions.
     10.000  T0              Background potential temperature (C) constant.
     35.000  S0              Background salinity (PSU) constant.
      1.000  gamma2          Slipperiness variable: free-slip (1.0) or 
                                                    no-slip (-1.0).

 Output/Input Files:

             Output Restart File:  ocean_rst.nc

 Activated C-preprocessing Options:

 BENCHMARK           Benchmark Test, Idealized Southern Ocean, Large Grid
 ALBEDO              Shortwave radiation from albedo equation.
 ANA_BSFLUX          Analytical kinematic bottom salinity flux.
 ANA_BTFLUX          Analytical kinematic bottom temperature flux.
 ANA_CLOUD           Analytical cloud fraction.
 ANA_GRID            Analytical grid set-up.
 ANA_HUMIDITY        Analytical surface air humidity.
 ANA_INITIAL         Analytical initial conditions.
 ANA_PAIR            Analytical surface air pressure.
 ANA_RAIN            Analytical rain fall rate.
 ANA_SRFLUX          Analytical kinematic shortwave radiation flux.
 ANA_SSFLUX          Analytical kinematic surface salinity flux.
 ANA_WINDS           Analytical surface wind components.
 ASSUMED_SHAPE       Using assumed-shape arrays.
 BULK_FLUXES         Surface bulk fluxes parametererization.
 CURVGRID            Orthogonal curvilinear grid.
 DJ_GRADPS           Parabolic Splines density Jacobian (Shchepetkin, 2002).
 DOUBLE_PRECISION    Double precision arithmetic.
 EW_PERIODIC         East-West periodic boundaries.
 LMD_CONVEC          LMD convective mixing due to shear instability.
 LMD_MIXING          Large/McWilliams/Doney interior mixing.
 LMD_NONLOCAL        LMD convective nonlocal transport.
 LMD_RIMIX           LMD diffusivity due to shear instability.
 LMD_SKPP            KPP surface boundary layer mixing.
 LONGWAVE            Compute net longwave radiation internally.
 MIX_GEO_TS          Mixing of tracers along geopotential surfaces.
 MIX_S_UV            Mixing of momentum along constant S-surfaces.
 NONLINEAR           Nonlinear Model.
 NONLIN_EOS          Nonlinear Equation of State for seawater.
 NORTHERN_WALL       Wall boundary at Northern edge.
 POWER_LAW           Power-law shape time-averaging barotropic filter.
 PROFILE             Time profiling activated .
 !RST_SINGLE         Double precision fields in restart NetCDF file.
 SALINITY            Using salinity.
 SOLAR_SOURCE        Solar Radiation Source Term.
 SOLVE3D             Solving 3D Primitive Equations.
 SOUTHERN_WALL       Wall boundary at Southern edge.
 SPLINES             Conservative parabolic spline reconstruction.
 SPHERICAL           Spherical grid configuration.
 TS_U3HADVECTION     Third-order upstream horizontal advection of tracers.
 TS_C4VADVECTION     Fourth-order centered vertical advection of tracers.
 TS_DIF2             Harmonic mixing of tracers.
 UV_ADV              Advection of momentum.
 UV_COR              Coriolis term.
 UV_U3HADVECTION     Third-order upstream horizontal advection of 3D momentum.
 UV_C4VADVECTION     Fourth-order centered vertical advection of momentum.
 UV_QDRAG            Quadratic bottom stress.
 UV_VIS2             Harmonic mixing of momentum.
 VAR_RHO_2D          Variable density barotropic mode.
Operating system error: Cannot allocate memory
Allocation would exceed memory limit

Error termination. Backtrace:
z  0000000000000000 ra 00000000000fc064 sp 00000001feff3550 gp 00000000001cd2e0
tp 0000000000299d60 t0 0000000000000000 t1 0000000000000000 t2 00000000000003ff
s0 0000000000000001 s1 000000000018e4f0 a0 0000000000000000 a1 000000000018e4f0
a2 0000000000000001 a3 0000000000000022 a4 ffffffffffffffff a5 ffffffffffffffff
a6 0000000000000000 a7 00000000000000de s2 0000000000002000 s3 0000000000000000
s4 00000001feff3708 s5 00000000000e55f0 s6 0000000000000202 s7 0000000000000028
s8 fffffffffd6a3920 s9 000000000295c6e2 sA 000000000029bca0 sB 0000000000000000
t3 ffffffffffffffff t4 000000000000000f t5 00000000000000df t6 0000000000000000
sr 00000000000000f9 pc 00000000000e5610 va 0000000000000005 insn       ffffffff
User load segfault @ 0x0000000000000005
cycle = 110961877
instret = 110961884
******* Resetting core **********
s117 commented 4 years ago

Just like what I did in #15, the problem sovled after increasing the memory size to 16GB.

$ spike -m16384 pk -c sroms_base.riscv-m64 < ocean_benchmark3.in

Requesting target memory 0x400000000
******* Resetting core ********** 
****Initializing the processor system****
******* Resetting core ********** 
******* Resetting core ********** 
****Initialization complete****
 Process Information:

 Model Input Parameters:  ROMS/TOMS version 3.2  

 -----------------------------------------------------------------------------

 Benchmark Test, Idealized Southern Ocean, Large Grid
 Version adapted for spec, built without netcdf

 Operating system : 
 CPU/hardware     : 
 Compiler system  : 
 Compiler command : 
 Compiler flags   : 

 SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : 405M

 Local Root    : 
 Header Dir    : 
 Header file   : benchmark.h
 Analytical Dir: 

 Resolution, Grid 01: 2048x0512x040,  Parallel Threads:  0,  Tiling: 000x000

 Physical Parameters, Grid: 01
 =============================

        100  ntimes          Number of timesteps for 3-D equations.
     50.000  dt              Timestep size (s) for 3-D equations.
         20  ndtfast         Number of timesteps for 2-D equations between
                               each 3D timestep.
          1  ERstr           Starting ensemble/perturbation run number.
          1  ERend           Ending ensemble/perturbation run number.
          0  nrrec           Number of restart records to read from disk.
          T  LcycleRST       Switch to recycle time-records in restart file.
       1000  nRST            Number of timesteps between the writing of data
                               into restart fields.
          1  ninfo           Number of timesteps between print of information
                               to standard output.
          T  ldefout         Switch to create a new output NetCDF file(s).
       1000  nHIS            Number of timesteps between the writing fields
                               into history file.
 5.0000E+02  tnu2(01)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 01: temp
 5.0000E+02  tnu2(02)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 02: salt
 5.0000E+03  visc2           Horizontal, harmonic mixing coefficient (m2/s)
                               for momentum.
 1.0000E-05  Akt_bak(01)     Background vertical mixing coefficient (m2/s)
                               for tracer 01: temp
 1.0000E-05  Akt_bak(02)     Background vertical mixing coefficient (m2/s)
                               for tracer 02: salt
 1.0000E-04  Akv_bak         Background vertical mixing coefficient (m2/s)
                               for momentum.
 3.0000E-04  rdrg            Linear bottom drag coefficient (m/s).
 3.0000E-03  rdrg2           Quadratic bottom drag coefficient.
 2.0000E-02  Zob             Bottom roughness (m).
 1.0000E+01  blk_ZQ          Height (m) of surface air humidity measurement.
 1.0000E+01  blk_ZT          Height (m) of surface air temperature measurement.
 1.0000E+01  blk_ZW          Height (m) of surface winds measurement.
          1  lmd_Jwt         Jerlov water type.
          1  Vtransform      S-coordinate transformation equation.
          1  Vstretching     S-coordinate stretching function.
 5.0000E+00  theta_s         S-coordinate surface control parameter.
 4.0000E-01  theta_b         S-coordinate bottom  control parameter.
    200.000  Tcline          S-coordinate surface/bottom layer width (m) used
                               in vertical coordinate stretching.
   1025.000  rho0            Mean density (kg/m3) for Boussinesq approximation.
      0.000  dstart          Time-stamp assigned to model initialization (days).
       0.00  time_ref        Reference time for units attribute (yyyymmdd.dd)
 0.0000E+00  Tnudg(01)       Nudging/relaxation time scale (days)
                               for tracer 01: temp
 0.0000E+00  Tnudg(02)       Nudging/relaxation time scale (days)
                               for tracer 02: salt
 0.0000E+00  Znudg           Nudging/relaxation time scale (days)
                               for free-surface.
 0.0000E+00  M2nudg          Nudging/relaxation time scale (days)
                               for 2D momentum.
 0.0000E+00  M3nudg          Nudging/relaxation time scale (days)
                               for 3D momentum.
 0.0000E+00  obcfac          Factor between passive and active
                               open boundary conditions.
     10.000  T0              Background potential temperature (C) constant.
     35.000  S0              Background salinity (PSU) constant.
      1.000  gamma2          Slipperiness variable: free-slip (1.0) or 
                                                    no-slip (-1.0).

 Output/Input Files:

             Output Restart File:  ocean_rst.nc

 Activated C-preprocessing Options:

 BENCHMARK           Benchmark Test, Idealized Southern Ocean, Large Grid
 ALBEDO              Shortwave radiation from albedo equation.
 ANA_BSFLUX          Analytical kinematic bottom salinity flux.
 ANA_BTFLUX          Analytical kinematic bottom temperature flux.
 ANA_CLOUD           Analytical cloud fraction.
 ANA_GRID            Analytical grid set-up.
 ANA_HUMIDITY        Analytical surface air humidity.
 ANA_INITIAL         Analytical initial conditions.
 ANA_PAIR            Analytical surface air pressure.
 ANA_RAIN            Analytical rain fall rate.
 ANA_SRFLUX          Analytical kinematic shortwave radiation flux.
 ANA_SSFLUX          Analytical kinematic surface salinity flux.
 ANA_WINDS           Analytical surface wind components.
 ASSUMED_SHAPE       Using assumed-shape arrays.
 BULK_FLUXES         Surface bulk fluxes parametererization.
 CURVGRID            Orthogonal curvilinear grid.
 DJ_GRADPS           Parabolic Splines density Jacobian (Shchepetkin, 2002).
 DOUBLE_PRECISION    Double precision arithmetic.
 EW_PERIODIC         East-West periodic boundaries.
 LMD_CONVEC          LMD convective mixing due to shear instability.
 LMD_MIXING          Large/McWilliams/Doney interior mixing.
 LMD_NONLOCAL        LMD convective nonlocal transport.
 LMD_RIMIX           LMD diffusivity due to shear instability.
 LMD_SKPP            KPP surface boundary layer mixing.
 LONGWAVE            Compute net longwave radiation internally.
 MIX_GEO_TS          Mixing of tracers along geopotential surfaces.
 MIX_S_UV            Mixing of momentum along constant S-surfaces.
 NONLINEAR           Nonlinear Model.
 NONLIN_EOS          Nonlinear Equation of State for seawater.
 NORTHERN_WALL       Wall boundary at Northern edge.
 POWER_LAW           Power-law shape time-averaging barotropic filter.
 PROFILE             Time profiling activated .
 !RST_SINGLE         Double precision fields in restart NetCDF file.
 SALINITY            Using salinity.
 SOLAR_SOURCE        Solar Radiation Source Term.
 SOLVE3D             Solving 3D Primitive Equations.
 SOUTHERN_WALL       Wall boundary at Southern edge.
 SPLINES             Conservative parabolic spline reconstruction.
 SPHERICAL           Spherical grid configuration.
 TS_U3HADVECTION     Third-order upstream horizontal advection of tracers.
 TS_C4VADVECTION     Fourth-order centered vertical advection of tracers.
 TS_DIF2             Harmonic mixing of tracers.
 UV_ADV              Advection of momentum.
 UV_COR              Coriolis term.
 UV_U3HADVECTION     Third-order upstream horizontal advection of 3D momentum.
 UV_C4VADVECTION     Fourth-order centered vertical advection of momentum.
 UV_QDRAG            Quadratic bottom stress.
 UV_VIS2             Harmonic mixing of momentum.
 VAR_RHO_2D          Variable density barotropic mode.

 INITIAL: Configuring and initializing forward nonlinear model ...

 Vertical S-coordinate System: 

 level   S-coord     Cs-curve          at_hmin  over_slope     at_hmax

    40   0.0000000  -0.0000000          -0.000      -0.000      -0.000
    39  -0.0250000  -0.0017775          -5.533      -8.644     -11.754
    38  -0.0500000  -0.0037835         -11.135     -17.756     -24.377
    37  -0.0750000  -0.0060921         -16.828     -27.489     -38.150
    36  -0.1000000  -0.0087922         -22.638     -38.024     -53.410
    35  -0.1250000  -0.0119914         -28.597     -49.582     -70.567
    34  -0.1500000  -0.0158197         -34.746     -62.430     -90.115
    33  -0.1750000  -0.0204330         -41.130     -76.888    -112.646
    32  -0.2000000  -0.0260168         -47.805     -93.334    -138.864
    31  -0.2250000  -0.0327879         -54.836    -112.215    -169.594
    30  -0.2500000  -0.0409944         -62.298    -134.039    -205.779
    29  -0.2750000  -0.0509117         -70.274    -159.369    -248.465
    28  -0.3000000  -0.0628318         -78.850    -188.805    -298.761
    27  -0.3250000  -0.0770446         -88.113    -222.941    -357.769
    26  -0.3500000  -0.0938098         -98.143    -262.310    -426.477
    25  -0.3750000  -0.1133185        -108.996    -307.303    -505.610
    24  -0.4000000  -0.1356491        -120.695    -358.081    -595.466
    23  -0.4250000  -0.1607245        -133.217    -414.485    -695.753
    22  -0.4500000  -0.1882840        -146.485    -475.982    -805.479
    21  -0.4750000  -0.2178817        -160.365    -541.658    -922.951
    20  -0.5000000  -0.2489214        -174.676    -610.289   -1045.901
    19  -0.5250000  -0.2807264        -189.218    -680.489   -1171.760
    18  -0.5500000  -0.3126323        -203.790    -750.896   -1298.003
    17  -0.5750000  -0.3440788        -218.224    -820.362   -1422.499
    16  -0.6000000  -0.3746809        -232.404    -888.096   -1543.788
    15  -0.6250000  -0.4042644        -246.279    -953.742   -1661.205
    14  -0.6500000  -0.4328656        -259.860   -1017.374   -1774.889
    13  -0.6750000  -0.4607052        -273.212   -1079.446   -1885.680
    12  -0.7000000  -0.4881475        -286.444   -1140.702   -1994.960
    11  -0.7250000  -0.5156591        -299.698   -1202.101   -2104.505
    10  -0.7500000  -0.5437741        -313.132   -1264.737   -2216.342
     9  -0.7750000  -0.5730691        -326.921   -1329.792   -2332.663
     8  -0.8000000  -0.6041494        -341.245   -1398.506   -2455.768
     7  -0.8250000  -0.6376434        -356.293   -1472.169   -2588.045
     6  -0.8500000  -0.6742046        -372.261   -1552.119   -2731.978
     5  -0.8750000  -0.7145184        -389.356   -1639.763   -2890.170
     4  -0.9000000  -0.7593114        -407.793   -1736.588   -3065.383
     3  -0.9250000  -0.8093643        -427.809   -1844.197   -3260.585
     2  -0.9500000  -0.8655258        -449.658   -1964.328   -3478.998
     1  -0.9750000  -0.9287272        -473.618   -2098.891   -3724.163
     0  -1.0000000  -1.0000000        -500.000   -2250.000   -4000.000

 Time Splitting Weights: ndtfast =  20    nfast =  29

    Primary            Secondary            Accumulated to Current Step

  1 -0.0009651193358779  0.0500000000000000 -0.0009651193358779  0.0500000000000000
  2 -0.0013488780126037  0.0500482559667939 -0.0023139973484816  0.1000482559667939
  3 -0.0011514592651644  0.0501156998674241 -0.0034654566136460  0.1501639558342180
  4 -0.0003735756740661  0.0501732728306823 -0.0038390322877121  0.2003372286649003
  5  0.0009829200513762  0.0501919516143856 -0.0028561122363360  0.2505291802792859
  6  0.0029141799764308  0.0501428056118168  0.0000580677400949  0.3006719858911027
  7  0.0054132615310267  0.0499970966129953  0.0054713292711215  0.3506690825040980
  8  0.0084687837865132  0.0497264335364439  0.0139401130576347  0.4003955160405419
  9  0.0120633394191050  0.0493029943471183  0.0260034524767397  0.4496985103876602
 10  0.0161716623600090  0.0486998273761630  0.0421751148367487  0.4983983377638232
 11  0.0207585511322367  0.0478912442581626  0.0629336659689854  0.5462895820219857
 12  0.0257765478740990  0.0468533167015507  0.0887102138430844  0.5931428987235364
 13  0.0311633730493853  0.0455644893078458  0.1198735868924697  0.6387073880313822
 14  0.0368391158442262  0.0440063206553765  0.1567127027366959  0.6827137086867586
 15  0.0427031802506397  0.0421643648631652  0.1994158829873356  0.7248780735499238
 16  0.0486309868367616  0.0400292058506332  0.2480468698240972  0.7649072794005570
 17  0.0544704302037592  0.0375976565087951  0.3025173000278564  0.8025049359093521
 18  0.0600380921294286  0.0348741349986072  0.3625553921572849  0.8373790709079593
 19  0.0651152103984763  0.0318722303921357  0.4276706025557612  0.8692513013000950
 20  0.0694434033194839  0.0286164698722119  0.4971140058752452  0.8978677711723069
 21  0.0727201499285570  0.0251442997062377  0.5698341558038021  0.9230120708785446
 22  0.0745940258796570  0.0215082922098099  0.6444281816834592  0.9445203630883545
 23  0.0746596950216180  0.0177785909158270  0.7190878767050772  0.9622989540041815
 24  0.0724526566618460  0.0140456061647461  0.7915405333669232  0.9763445601689277
 25  0.0674437485167025  0.0104229733316538  0.8589842818836257  0.9867675335005816
 26  0.0590334053485720  0.0070507859058187  0.9180176872321977  0.9938183194064003
 27  0.0465456732896124  0.0040991156383901  0.9645633605218101  0.9979174350447905
 28  0.0292219798521903  0.0017718319739095  0.9937853403740005  0.9996892670187000
 29  0.0062146596259994  0.0003107329813000  0.9999999999999999  1.0000000000000000

 ndtfast, nfast =   20  29   nfast/ndtfast = 1.45000

 Centers of gravity and integrals (values must be 1, 1, approx 1/2, 1, 1):

    1.000000000000 1.060707743385 0.530353871693 1.000000000000 1.000000000000

 Power filter parameters, Fgamma, gamma =  0.28400   0.14200

 Minimum X-grid spacing, DXmin =  6.67918871E+00 km
 Maximum X-grid spacing, DXmax =  1.25696406E+01 km
 Minimum Y-grid spacing, DYmin =  4.34376658E+00 km
 Maximum Y-grid spacing, DYmax =  4.34376658E+00 km
 Minimum Z-grid spacing, DZmin =  5.53323611E+00 m
 Maximum Z-grid spacing, DZmax =  2.75836519E+02 m

 Minimum barotropic Courant Number =  4.69783444E-02
 Maximum barotropic Courant Number =  1.32301112E-01
 Maximum Coriolis   Courant Number =  6.85319684E-03

 Maximum grid stiffness ratios:  rx0 =   3.966042E-01 (Beckmann and Haidvogel)
                                 rx1 =   1.534788E+01 (Haney)

 Initial basin volumes: TotVolume =  1.6599435966E+17 m3
                        MinVolume =  1.6083586258E+08 m3
                        MaxVolume =  1.5048327667E+10 m3
                          Max/Min =  9.3563260243E+01

NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 00000001 - 00000100)

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME

      0     0 00:00:00  0.000000E+00  1.963034E+04  1.963034E+04  1.659944E+17
      1     0 00:00:50  1.538721E-09  1.963034E+04  1.963034E+04  1.659944E+17
      2     0 00:01:40  6.087962E-09  1.963034E+04  1.963034E+04  1.659944E+17
      3     0 00:02:30  1.362528E-08  1.963034E+04  1.963034E+04  1.659944E+17
      4     0 00:03:20  2.413734E-08  1.963034E+04  1.963034E+04  1.659944E+17
      5     0 00:04:10  3.760411E-08  1.963034E+04  1.963034E+04  1.659944E+17
      6     0 00:05:00  5.400755E-08  1.963034E+04  1.963034E+04  1.659944E+17
      7     0 00:05:50  7.333129E-08  1.963034E+04  1.963034E+04  1.659944E+17
      8     0 00:06:40  9.555774E-08  1.963034E+04  1.963034E+04  1.659944E+17
      9     0 00:07:30  1.206714E-07  1.963034E+04  1.963034E+04  1.659944E+17
     10     0 00:08:20  1.486556E-07  1.963034E+04  1.963034E+04  1.659944E+17
     11     0 00:09:10  1.794954E-07  1.963034E+04  1.963034E+04  1.659944E+17
     12     0 00:10:00  2.131747E-07  1.963034E+04  1.963034E+04  1.659944E+17
     13     0 00:10:50  2.496782E-07  1.963034E+04  1.963034E+04  1.659944E+17
     14     0 00:11:40  2.863378E-07  1.963034E+04  1.963034E+04  1.659944E+17
     15     0 00:12:30  3.236174E-07  1.963034E+04  1.963034E+04  1.659944E+17
     16     0 00:13:20  3.618818E-07  1.963034E+04  1.963034E+04  1.659944E+17
     17     0 00:14:10  4.011100E-07  1.963034E+04  1.963034E+04  1.659944E+17
     18     0 00:15:00  4.413145E-07  1.963034E+04  1.963034E+04  1.659944E+17
     19     0 00:15:50  4.823935E-07  1.963034E+04  1.963034E+04  1.659944E+17
     20     0 00:16:40  5.244001E-07  1.963034E+04  1.963034E+04  1.659944E+17
     21     0 00:17:30  5.672672E-07  1.963034E+04  1.963034E+04  1.659944E+17
     22     0 00:18:20  6.109431E-07  1.963034E+04  1.963034E+04  1.659944E+17
     23     0 00:19:10  6.555108E-07  1.963034E+04  1.963034E+04  1.659944E+17
     24     0 00:20:00  7.008151E-07  1.963034E+04  1.963034E+04  1.659944E+17
     25     0 00:20:50  7.469519E-07  1.963034E+04  1.963034E+04  1.659944E+17
     26     0 00:21:40  7.939017E-07  1.963034E+04  1.963034E+04  1.659944E+17
     27     0 00:22:30  8.417621E-07  1.963034E+04  1.963034E+04  1.659944E+17
     28     0 00:23:20  8.904065E-07  1.963034E+04  1.963034E+04  1.659944E+17
     29     0 00:24:10  9.398310E-07  1.963034E+04  1.963034E+04  1.659944E+17
     30     0 00:25:00  9.901400E-07  1.963034E+04  1.963034E+04  1.659944E+17
     31     0 00:25:50  1.041224E-06  1.963034E+04  1.963034E+04  1.659944E+17
     32     0 00:26:40  1.093189E-06  1.963034E+04  1.963034E+04  1.659944E+17
     33     0 00:27:30  1.145938E-06  1.963034E+04  1.963034E+04  1.659944E+17
     34     0 00:28:20  1.199311E-06  1.963034E+04  1.963034E+04  1.659944E+17
     35     0 00:29:10  1.253430E-06  1.963034E+04  1.963034E+04  1.659944E+17
     36     0 00:30:00  1.308301E-06  1.963034E+04  1.963034E+04  1.659944E+17
     37     0 00:30:50  1.363900E-06  1.963034E+04  1.963034E+04  1.659944E+17
     38     0 00:31:40  1.420199E-06  1.963034E+04  1.963034E+04  1.659944E+17
     39     0 00:32:30  1.477194E-06  1.963034E+04  1.963034E+04  1.659944E+17
     40     0 00:33:20  1.534860E-06  1.963034E+04  1.963034E+04  1.659944E+17
     41     0 00:34:10  1.593173E-06  1.963034E+04  1.963034E+04  1.659944E+17
     42     0 00:35:00  1.652108E-06  1.963034E+04  1.963034E+04  1.659944E+17
     43     0 00:35:50  1.711640E-06  1.963034E+04  1.963034E+04  1.659944E+17
     44     0 00:36:40  1.771770E-06  1.963034E+04  1.963034E+04  1.659944E+17
     45     0 00:37:30  1.832452E-06  1.963034E+04  1.963034E+04  1.659944E+17
     46     0 00:38:20  1.893690E-06  1.963034E+04  1.963034E+04  1.659944E+17
     47     0 00:39:10  1.955464E-06  1.963034E+04  1.963034E+04  1.659944E+17
     48     0 00:40:00  2.017728E-06  1.963034E+04  1.963034E+04  1.659944E+17
     49     0 00:40:50  2.080492E-06  1.963034E+04  1.963034E+04  1.659944E+17
     50     0 00:41:40  2.143816E-06  1.963034E+04  1.963034E+04  1.659944E+17
     51     0 00:42:30  2.207602E-06  1.963034E+04  1.963034E+04  1.659944E+17
     52     0 00:43:20  2.271836E-06  1.963034E+04  1.963034E+04  1.659944E+17
     53     0 00:44:10  2.336506E-06  1.963034E+04  1.963034E+04  1.659944E+17
     54     0 00:45:00  2.401673E-06  1.963034E+04  1.963034E+04  1.659944E+17
     55     0 00:45:50  2.467245E-06  1.963034E+04  1.963034E+04  1.659944E+17
     56     0 00:46:40  2.533215E-06  1.963034E+04  1.963034E+04  1.659944E+17
     57     0 00:47:30  2.599642E-06  1.963034E+04  1.963034E+04  1.659944E+17
     58     0 00:48:20  2.666443E-06  1.963034E+04  1.963034E+04  1.659944E+17
     59     0 00:49:10  2.733644E-06  1.963034E+04  1.963034E+04  1.659944E+17
     60     0 00:50:00  2.801273E-06  1.963034E+04  1.963034E+04  1.659944E+17
     61     0 00:50:50  2.869250E-06  1.963034E+04  1.963034E+04  1.659944E+17
     62     0 00:51:40  2.937570E-06  1.963034E+04  1.963034E+04  1.659944E+17
     63     0 00:52:30  3.006304E-06  1.963034E+04  1.963034E+04  1.659944E+17
     64     0 00:53:20  3.075357E-06  1.963034E+04  1.963034E+04  1.659944E+17
     65     0 00:54:10  3.144741E-06  1.963034E+04  1.963034E+04  1.659944E+17
     66     0 00:55:00  3.214396E-06  1.963034E+04  1.963034E+04  1.659944E+17
     67     0 00:55:50  3.284436E-06  1.963034E+04  1.963034E+04  1.659944E+17
     68     0 00:56:40  3.354747E-06  1.963034E+04  1.963034E+04  1.659944E+17
     69     0 00:57:30  3.425378E-06  1.963034E+04  1.963034E+04  1.659944E+17
     70     0 00:58:20  3.496321E-06  1.963034E+04  1.963034E+04  1.659944E+17
     71     0 00:59:10  3.567524E-06  1.963034E+04  1.963034E+04  1.659944E+17
     72     0 01:00:00  3.639069E-06  1.963034E+04  1.963034E+04  1.659944E+17
     73     0 01:00:50  3.710875E-06  1.963034E+04  1.963034E+04  1.659944E+17
     74     0 01:01:40  3.782992E-06  1.963034E+04  1.963034E+04  1.659944E+17
     75     0 01:02:30  3.855366E-06  1.963034E+04  1.963034E+04  1.659944E+17
     76     0 01:03:20  3.928014E-06  1.963034E+04  1.963034E+04  1.659944E+17
     77     0 01:04:10  4.000959E-06  1.963034E+04  1.963034E+04  1.659944E+17
     78     0 01:05:00  4.074114E-06  1.963034E+04  1.963034E+04  1.659944E+17
     79     0 01:05:50  4.147571E-06  1.963034E+04  1.963034E+04  1.659944E+17
     80     0 01:06:40  4.221227E-06  1.963034E+04  1.963034E+04  1.659944E+17
     81     0 01:07:30  4.295169E-06  1.963034E+04  1.963034E+04  1.659944E+17
     82     0 01:08:20  4.369321E-06  1.963034E+04  1.963034E+04  1.659944E+17
     83     0 01:09:10  4.443723E-06  1.963034E+04  1.963034E+04  1.659944E+17
     84     0 01:10:00  4.518366E-06  1.963034E+04  1.963034E+04  1.659944E+17
     85     0 01:10:50  4.593213E-06  1.963034E+04  1.963034E+04  1.659944E+17
     86     0 01:11:40  4.668325E-06  1.963034E+04  1.963034E+04  1.659944E+17
     87     0 01:12:30  4.743595E-06  1.963034E+04  1.963034E+04  1.659944E+17
     88     0 01:13:20  4.819109E-06  1.963034E+04  1.963034E+04  1.659944E+17
     89     0 01:14:10  4.894794E-06  1.963034E+04  1.963034E+04  1.659944E+17
     90     0 01:15:00  4.970707E-06  1.963034E+04  1.963034E+04  1.659944E+17
     91     0 01:15:50  5.046849E-06  1.963034E+04  1.963034E+04  1.659944E+17
     92     0 01:16:40  5.123144E-06  1.963034E+04  1.963034E+04  1.659944E+17
     93     0 01:17:30  5.199657E-06  1.963034E+04  1.963034E+04  1.659944E+17
     94     0 01:18:20  5.276310E-06  1.963034E+04  1.963034E+04  1.659944E+17
     95     0 01:19:10  5.353176E-06  1.963034E+04  1.963034E+04  1.659944E+17
     96     0 01:20:00  5.430248E-06  1.963034E+04  1.963034E+04  1.659944E+17
     97     0 01:20:50  5.507456E-06  1.963034E+04  1.963034E+04  1.659944E+17
     98     0 01:21:40  5.584857E-06  1.963034E+04  1.963034E+04  1.659944E+17
     99     0 01:22:30  5.662435E-06  1.963034E+04  1.963034E+04  1.659944E+17
    100     0 01:23:20  5.740154E-06  1.963034E+04  1.963034E+04  1.659944E+17

 Elapsed CPU time (seconds):

 ROMS/TOMS - Output NetCDF summary for Grid 01:

 ROMS/TOMS: DONE... 
cycle = 23144204799685
instret = 23144204799692
******* Resetting core **********