search

underworldcode / UWGeodynamics

Underworld Geodynamics
Other
81 stars 32 forks source link

Restart is stuck within `Time Integration - 7.15256e-06 [min] / 0.561747 [max] (secs)` #239

Closed tyszwh closed 3 years ago

tyszwh commented 3 years ago

Hi, Dr Romain

I have a problem with restart fucntion. Fisrtly I ran the model with mpirun - np 8 python 3 xx.py And then I restart it with the same command. It seems that the restart is normal but it is stcuk within Time Integration - 7.15256e-06 [min] / 0.561747 [max] (secs)

The output is shown below. 


    Local offset of rank 0: 0x0
    Local range of rank 0: 32x16
In func WeightsCalculator_CalculateAll(): for swarm "AOD3K4U9__swarm"
    done 33% (171 cells)...
    done 67% (342 cells)...
    done 100% (512 cells)...
WeightsCalculator_CalculateAll(): finished update of weights for swarm "AOD3K4U9__swarm"
colourscolourbar : Invalid property name
colourscolourbar : Invalid property name
64 32
45.0
1.5
Length, km =  3000.0 kilometer
Time, Myr =  60.0 megayear
Pressure, MPa =  97119.0 megapascal
Temperature, K =  1300.0 kelvin
Velocity, cm/yr =  5.0 centimeter / year
Viscosity, Pa S =  1.8389055326399998e+26 pascal * second

------------------------------------
Restarting Model from Step 15 at Time = 22499999.99999996 year

(2021-07-21 08:07:16)
----------------------------------
Mesh loaded(2021-07-21 08:07:16)
Swarm loaded(2021-07-21 08:07:17)
In func WeightsCalculator_CalculateAll(): for swarm "T6C20BYN__swarm"
    done 33% (171 cells)...
    done 67% (342 cells)...
    done 100% (512 cells)...
WeightsCalculator_CalculateAll(): finished update of weights for swarm "T6C20BYN__swarm"
pressureField loaded(2021-07-21 08:07:17)
velocityField loaded(2021-07-21 08:07:17)
materialField loaded(2021-07-21 08:07:17)
plasticStrain loaded(2021-07-21 08:07:17)
timeField loaded(2021-07-21 08:07:17)
temperature loaded(2021-07-21 08:07:17)
FSE_Crust loaded(2021-07-21 08:07:20)
Running with UWGeodynamics version 2.10.5
In SystemLinearEquations_NonLinearExecute

Non linear solver - iteration 0
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001654 seconds

SROpGenerator_SimpleFinestLevel: time = 1.16706e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.61953e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.52859e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.92063e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.75135e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002804 secs 
  RHS V Solve:            = 0.0292 secs / 107 its
  Pressure Solve:         = 0.1498 secs / 9 its
  Final V Solve:          = 0.0202 secs / 76 its

  Total BSSCR Linear solve time: 0.214194 seconds

Linear solver (HOXYKF5B__system-execute), solution time 2.152960e-01 (secs)
Non linear solver - iteration 1
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001636 seconds

SROpGenerator_SimpleFinestLevel: time = 1.14894e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.57184e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.44038e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.93970e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.83957e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002958 secs 
  RHS V Solve:            = 0.03873 secs / 143 its
  Pressure Solve:         = 0.184 secs / 10 its
  Final V Solve:          = 0.0267 secs / 101 its

  Total BSSCR Linear solve time: 0.264317 seconds

Linear solver (HOXYKF5B__system-execute), solution time 2.648430e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 1 of 50 - Residual 0.17134 - Tolerance = 0.01
Non linear solver - Residual 1.71342722e-01; Tolerance 1.0000e-02 - Not converged - 9.336109e-01 (secs)

Non linear solver - iteration 2
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001715 seconds

SROpGenerator_SimpleFinestLevel: time = 1.14799e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.55992e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.44038e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.88963e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.67982e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002933 secs 
  RHS V Solve:            = 0.04427 secs / 163 its
  Pressure Solve:         = 0.2204 secs / 11 its
  Final V Solve:          = 0.02986 secs / 113 its

  Total BSSCR Linear solve time: 0.309468 seconds

Linear solver (HOXYKF5B__system-execute), solution time 3.100150e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 2 of 50 - Residual 0.19908 - Tolerance = 0.01
Non linear solver - Residual 1.99079032e-01; Tolerance 1.0000e-02 - Not converged - 1.510276e+00 (secs)

Non linear solver - iteration 3
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001642 seconds

SROpGenerator_SimpleFinestLevel: time = 1.15013e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.49078e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.46899e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.95877e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.82049e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002949 secs 
  RHS V Solve:            = 0.04735 secs / 175 its
  Pressure Solve:         = 0.2394 secs / 11 its
  Final V Solve:          = 0.03632 secs / 136 its

  Total BSSCR Linear solve time: 0.337974 seconds

Linear solver (HOXYKF5B__system-execute), solution time 3.384960e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 3 of 50 - Residual 0.2968 - Tolerance = 0.01
Non linear solver - Residual 2.96801757e-01; Tolerance 1.0000e-02 - Not converged - 2.102279e+00 (secs)

Non linear solver - iteration 4
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001638 seconds

SROpGenerator_SimpleFinestLevel: time = 1.15204e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.58138e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.44038e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.83957e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.64883e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002936 secs 
  RHS V Solve:            = 0.04898 secs / 180 its
  Pressure Solve:         = 0.2587 secs / 11 its
  Final V Solve:          = 0.0473 secs / 177 its

  Total BSSCR Linear solve time: 0.369785 seconds

Linear solver (HOXYKF5B__system-execute), solution time 3.703091e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 4 of 50 - Residual 0.2844 - Tolerance = 0.01
Non linear solver - Residual 2.84403140e-01; Tolerance 1.0000e-02 - Not converged - 2.725441e+00 (secs)

Non linear solver - iteration 5
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001646 seconds

SROpGenerator_SimpleFinestLevel: time = 1.15490e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.48840e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.43084e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.98977e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.83003e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002958 secs 
  RHS V Solve:            = 0.05494 secs / 205 its
  Pressure Solve:         = 0.2609 secs / 11 its
  Final V Solve:          = 0.05762 secs / 218 its

  Total BSSCR Linear solve time: 0.388425 seconds

Linear solver (HOXYKF5B__system-execute), solution time 3.889608e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 5 of 50 - Residual 0.20076 - Tolerance = 0.01
Non linear solver - Residual 2.00763788e-01; Tolerance 1.0000e-02 - Not converged - 3.368545e+00 (secs)

Non linear solver - iteration 6
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001645 seconds

SROpGenerator_SimpleFinestLevel: time = 1.14799e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.62906e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.39985e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.83003e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.66075e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002926 secs 
  RHS V Solve:            = 0.06254 secs / 231 its
  Pressure Solve:         = 0.258 secs / 11 its
  Final V Solve:          = 0.06484 secs / 241 its

  Total BSSCR Linear solve time: 0.400095 seconds

Linear solver (HOXYKF5B__system-execute), solution time 4.006231e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 6 of 50 - Residual 0.11775 - Tolerance = 0.01
Non linear solver - Residual 1.17751964e-01; Tolerance 1.0000e-02 - Not converged - 4.023182e+00 (secs)

Non linear solver - iteration 7
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001645 seconds

SROpGenerator_SimpleFinestLevel: time = 1.14703e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.50032e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.42131e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.98977e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.83957e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002942 secs 
  RHS V Solve:            = 0.06495 secs / 243 its
  Pressure Solve:         = 0.2603 secs / 11 its
  Final V Solve:          = 0.06776 secs / 254 its

  Total BSSCR Linear solve time: 0.407889 seconds

Linear solver (HOXYKF5B__system-execute), solution time 4.084179e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 7 of 50 - Residual 0.064721 - Tolerance = 0.01
Non linear solver - Residual 6.47213053e-02; Tolerance 1.0000e-02 - Not converged - 4.684452e+00 (secs)

Non linear solver - iteration 8
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001645 seconds

SROpGenerator_SimpleFinestLevel: time = 1.15514e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.57899e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.42131e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.87771e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.67982e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002944 secs 
  RHS V Solve:            = 0.06458 secs / 237 its
  Pressure Solve:         = 0.2657 secs / 11 its
  Final V Solve:          = 0.06799 secs / 255 its

  Total BSSCR Linear solve time: 0.413133 seconds

Linear solver (HOXYKF5B__system-execute), solution time 4.136591e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 8 of 50 - Residual 0.039304 - Tolerance = 0.01
Non linear solver - Residual 3.93043042e-02; Tolerance 1.0000e-02 - Not converged - 5.350783e+00 (secs)

Non linear solver - iteration 9
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001649 seconds

SROpGenerator_SimpleFinestLevel: time = 1.14799e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.50032e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.41177e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.97070e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.84195e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002946 secs 
  RHS V Solve:            = 0.06484 secs / 239 its
  Pressure Solve:         = 0.2659 secs / 11 its
  Final V Solve:          = 0.06829 secs / 255 its

  Total BSSCR Linear solve time: 0.413849 seconds

Linear solver (HOXYKF5B__system-execute), solution time 4.143710e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 9 of 50 - Residual 0.026599 - Tolerance = 0.01
Non linear solver - Residual 2.65993306e-02; Tolerance 1.0000e-02 - Not converged - 6.019332e+00 (secs)

Non linear solver - iteration 10
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001643 seconds

SROpGenerator_SimpleFinestLevel: time = 1.15204e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.59092e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.44038e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.83957e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.68221e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002931 secs 
  RHS V Solve:            = 0.06621 secs / 244 its
  Pressure Solve:         = 0.2647 secs / 11 its
  Final V Solve:          = 0.06757 secs / 254 its

  Total BSSCR Linear solve time: 0.413337 seconds

Linear solver (HOXYKF5B__system-execute), solution time 4.138610e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 10 of 50 - Residual 0.018915 - Tolerance = 0.01
Non linear solver - Residual 1.89153443e-02; Tolerance 1.0000e-02 - Not converged - 6.686041e+00 (secs)

Non linear solver - iteration 11
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001633 seconds

SROpGenerator_SimpleFinestLevel: time = 1.15895e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.50985e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.41892e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.97070e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.82049e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002961 secs 
  RHS V Solve:            = 0.06588 secs / 243 its
  Pressure Solve:         = 0.265 secs / 11 its
  Final V Solve:          = 0.06897 secs / 258 its

  Total BSSCR Linear solve time: 0.414576 seconds

Linear solver (HOXYKF5B__system-execute), solution time 4.150991e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 11 of 50 - Residual 0.013917 - Tolerance = 0.01
Non linear solver - Residual 1.39174766e-02; Tolerance 1.0000e-02 - Not converged - 7.354828e+00 (secs)

Non linear solver - iteration 12
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001642 seconds

SROpGenerator_SimpleFinestLevel: time = 1.15013e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.61953e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.42846e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.85864e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.68936e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002941 secs 
  RHS V Solve:            = 0.06564 secs / 244 its
  Pressure Solve:         = 0.2639 secs / 11 its
  Final V Solve:          = 0.06716 secs / 256 its

  Total BSSCR Linear solve time: 0.411789 seconds

Linear solver (HOXYKF5B__system-execute), solution time 4.123149e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 12 of 50 - Residual 0.010203 - Tolerance = 0.01
Non linear solver - Residual 1.02032373e-02; Tolerance 1.0000e-02 - Not converged - 8.020811e+00 (secs)

Non linear solver - iteration 13
Linear solver (HOXYKF5B__system-execute) 

BSSCR -- Block Stokes Schur Compliment Reduction Solver 

-----  K2_GMG  ------

AUGMENTED LAGRANGIAN K2 METHOD - Penalty = 0.001000

    * K+p*K2 in time: 0.001638 seconds

SROpGenerator_SimpleFinestLevel: time = 1.15085e-03 
  [4] SROpGenerator_SimpleCoarserLevel: time = 5.55038e-04 
  [3] SROpGenerator_SimpleCoarserLevel: time = 3.40939e-04 
  [2] SROpGenerator_SimpleCoarserLevel: time = 2.96116e-04 
  [1] SROpGenerator_SimpleCoarserLevel: time = 2.85149e-04 
  Setting schur_pc to "gkgdiag" 

SCR Solver Summary:

  Multigrid setup:        = 0.002958 secs 
  RHS V Solve:            = 0.066 secs / 245 its
  Pressure Solve:         = 0.2647 secs / 11 its
  Final V Solve:          = 0.06887 secs / 258 its

  Total BSSCR Linear solve time: 0.414332 seconds

Linear solver (HOXYKF5B__system-execute), solution time 4.148641e-01 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 13 of 50 - Residual 0.0077222 - Tolerance = 0.01
Non linear solver - Residual 7.72216178e-03; Tolerance 1.0000e-02 - Converged - 8.689128e+00 (secs)

In func SystemLinearEquations_NonLinearExecute: Converged after 13 iterations.
Linear solver (1QJ8GOI5__system-execute) 
Linear solver (1QJ8GOI5__system-execute), solution time 7.307196e-02 (secs)
Time Integration
    2nd order:                 VJFVZZGE__integrand -    0.0235 [min] /    0.0436 [max] (secs)
Time Integration - 0.0235341 [min] / 0.0436299 [max] (secs)
Time Integration
    2nd order:                 67WKVXLQ__integrand -    0.0000 [min] /    0.5617 [max] (secs)
Time Integration - 7.15256e-06 [min] / 0.561747 [max] (secs)```
julesghub commented 3 years ago

Hi @tyszwh, Are you still having restart issues? If so can you share your python file with me. GitHub repo or gist? would be best. Also what version of UWGEO are you running?

rbeucher commented 3 years ago

Hi @tyszwh

Does not look like a restart issue to me... Can you tell us more?

R

tyszwh commented 3 years ago

Hi,@julesghub @rbeucher Thanks for your reply. The restart function works fine on my current computer. The problem occurs with another machine that is not currently in my hand. But I think this should not be a matter of code. However, I will check it when I'm there. I should feedback the progress of the problem in time, sorry for it. If there have further problems with it, I will reopen it. Thanks again.