search

ElmerCSC / elmer-elmag

Development repository for electromagnetics tutorial and verification cases
45 stars 20 forks source link

mpirun InductionHeating examples #10

Closed arvedes closed 1 year ago

arvedes commented 1 year ago

I just tried to run the InductionHeating cases with mpi on 4 cores:

@raback are there any known limitations of the applied solvers?

arvedes commented 1 year ago

Regarding the case_layer-conductivity.sif:

Output without mpi:

(...)
MapBodiesAndBCs: Maximum initial boundary index: 8
CreateIntersectionBCs: Number of intersection BCs to determine: 1
CreateIntersectionBCs: Number of candidate intersection parents: 12324
CreateIntersectionBCs: Allocated for 12 new 1D boundary elements!
ReleaseMeshFaceTables: Releasing number of faces: 459653
(...)

Output with mpi:

(...)
MapBodiesAndBCs: Maximum initial boundary index: 8
CreateIntersectionBCs: Number of intersection BCs to determine: 1
CreateIntersectionBCs: Number of candidate intersection parents: 3153
CreateIntersectionBCs: Could not find any additional interface elements!
ReleaseMeshFaceTables: Releasing number of faces: 115241
(...)
raback commented 1 year ago

Hi, this might not have been properly parallelized yet as it is very new feature. So this is what you get from partition 0. If you set Simulation::Max Output Partition = 32, for example, you should be able to see if something is created in other partitions.

arvedes commented 1 year ago

Hi Peter, indeed, the IntersectionBCs are allocated correctly

(...)
CreateIntersectionBCs: Part2: Number of intersection BCs to determine: 1
CreateIntersectionBCs: Part2: Number of candidate intersection parents: 3079
CreateIntersectionBCs: Part0: Number of intersection BCs to determine: 1
CreateIntersectionBCs: Part0: Number of candidate intersection parents: 3153
CreateIntersectionBCs: Part1: Number of intersection BCs to determine: 1
CreateIntersectionBCs: Part1: Number of candidate intersection parents: 3051
CreateIntersectionBCs: Part3: Could not find any additional interface elements!
ReleaseMeshFaceTables: Part3: Releasing number of faces: 116869
CreateIntersectionBCs: Part1: Allocated for 12 new 1D boundary elements!
(...)

Nevertheless, I think there is a problem with this feature as I don't see any current in the result. I just attach the complete log, maybe you can read something from it: layercond_mpi_fulloutput.log

Have there been any recent changes to that function? I used a solver from October 7th to run the case.

arvedes commented 1 year ago

Hi @raback, I tried to run the case again with a newer solver. The problem persists but I get a more elaborate error message:

(...)
OptimizeBandwidth: Part1: ---------------------------------------------------------
OptimizeBandwidth: Part1: Computing matrix structure for: heat equation...Part1: done.
OptimizeBandwidth: Part1: Half bandwidth without optimization: 10655
OptimizeBandwidth: Part1: 
OptimizeBandwidth: Part1: Bandwidth Optimization ...Part1: done.
OptimizeBandwidth: Part1: Half bandwidth after optimization: 1085
OptimizeBandwidth: Part1: ---------------------------------------------------------
Loading user function library: [ResultOutputSolve]...[ResultOutputSolver_Init]
Loading user function library: [ResultOutputSolve]...[ResultOutputSolver_bulk]
Loading user function library: [ResultOutputSolve]...[ResultOutputSolver]
ElmerSolver: Part1: Number of timesteps to be saved: 1
ListTagCount: Part1: Found number of normalized keywords: 1
CalculateEntityWeights: Part1: Computing weights for the mesh entities
ListSetParameters: Part1: Altered number of parameters: 1
 BC weight:           1   4.6640245897726904E-002
 BC weight:           2  0.41960528948285047     
 BC weight:           3   7.5012508973303668E-005
 BC weight:           4   7.5001405342127765E-005
 BC weight:           5   3.1059753213902308E-002
 BF weight:           1   5.6520471742070003E-004
 Body weight:           1   1.1257033199533879E-004
ListSetParameters: ListSetParameters:  Body weight:           2   5.6520471742070003E-004
 Body weight:           3   1.9945457442145341E-002
 Mat weight:           1   1.1257033199533879E-004
Part3: Altered number of parameters: 1
Part2: Altered number of parameters: 1
 Mat weight:           2   5.6520471742070003E-004
 Mat weight:           3   1.9945457442145341E-002
ListSetParameters: Part0: Altered number of parameters: 1

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fab4119ad21 in ???
#1  0x7fab41199ef5 in ???
#2  0x7fab40fcb08f in ???
#3  0x7fab41862ecf in __parallelutils_MOD_parallelinitmatrix
    at /home/elmer/elmerfem/fem/src/ParallelUtils.F90:145
#4  0x7fab416f72d3 in __mainutils_MOD_singlesolver
    at /home/elmer/elmerfem/fem/src/MainUtils.F90:5246
#5  0x7fab4170c538 in __mainutils_MOD_solveractivate
    at /home/elmer/elmerfem/fem/src/MainUtils.F90:5568
#6  0x7fab4170df7e in solvecoupled
    at /home/elmer/elmerfem/fem/src/MainUtils.F90:3199
#7  0x7fab4170f35e in __mainutils_MOD_solveequations
    at /home/elmer/elmerfem/fem/src/MainUtils.F90:2899
#8  0x7fab41985a13 in execsimulation
    at /home/elmer/elmerfem/fem/src/ElmerSolver.F90:3079
#9  0x7fab4198ce4a in elmersolver_
    at /home/elmer/elmerfem/fem/src/ElmerSolver.F90:607
#10  0x5609c50d93f5 in solver
    at /home/elmer/elmerfem/fem/src/Solver.F90:57
#11  0x5609c50d915e in main
    at /home/elmer/elmerfem/fem/src/Solver.F90:34

Here are the complete logs: layercond_mpi_allranks.log

Do you have any idea how I could fix that problem?

EDIT: I use a solver build from this commit: https://github.com/ElmerCSC/elmerfem/commit/d3b69300319f2aa696ca9cc08a9bb465008425fd

arvedes commented 1 year ago

I posted this issue here: http://www.elmerfem.org/forum/viewtopic.php?t=7916

raback commented 1 year ago

I seem to have issues with this even in serial. Does it work for you? Trying to narrow down where it broke down. Should add the feature set to the tests.

arvedes commented 1 year ago

You're right, I didn't try that again. With an older version (https://github.com/ElmerCSC/elmerfem/commit/0ff29c7213bfe81ab9f79de7650118d69bec1ab4, https://github.com/nemocrys/opencgs/blob/main/Docker/Dockerfile) it works well in serial but fails with mpi (as reported above: https://github.com/ElmerCSC/elmer-elmag/issues/10#issuecomment-1300083653).

arvedes commented 1 year ago

Update: I tried again with https://github.com/ElmerCSC/elmerfem/commit/893863e6194c74ffcb7e3d0640c766b6b7751b5e (13.03.2023)

juharu commented 1 year ago

Hi

Nice!

I also tried these: 

* >case_layer-conductivity.sif: gives trivial result with mpi (output here: [ https://github.com/ElmerCSC/elmer-elmag/files/11001715/layer-cond.log | layer-cond.log ] ) but works without mpi

Seems something not implemented for parallel usage ? Or maybe just buggy ? 

* case_circuit.sif: does not converge (neither with / without mpi)

These worked for me in the MGDynamics-Solver:

Linear System Complex = Logical True Linear System Solver = Iterative Linear System Iterative Method = GCR Linear System ILU Order = 0 Linear System preconditioning = Circuit Linear System Convergence Tolerance = 1.e-6 Linear System Max Iterations = 6000 Linear System Residual Output = 10

both in serial and in 4 task-mpi run.

Br, Juha

juharu commented 1 year ago

Linear System Iterative Method = GCR "BiCGstabL" also works, and is faster.

jvela018 commented 1 year ago

@arvedes and @juharu ,

I believe that in the plain circuit case the only issue was

Linear System Complex = Logical False

which should be

Linear System Complex = Logical True

The harmonic cases on the CircuitBuilder examples tend to have this because this applies to MacOS (which is how I built the examples). If you're working on Linux, you should set it to Linear System Complex = Logical True. Somehow on MacOS you still get the results for the imaginary part. I don't remember the reason...it's been a long time since I created those examples. However, I just tested the harmonic 3D open coil example and I that switch needs to be kept as False on MacOS and the results are correct.

BR,

Jon

juharu commented 1 year ago

Hi Jonathan 😀WellEnabling the incomplete LU on top of "Circuit"-preconditioner reallymade the case converge.But sure, we should use complex iterators for complex valued systems whenever possible.Br, Juha

juharu commented 1 year ago

 About the complex vs real iterators applied to complex valued systems: I did, way back when, arrange the linear system within elmersolver such that it can be thrown to both complex and real valued linear system solvers - not all solvers have complex arithmetic versions - notably most of the "direct" solvers.  The complex iterative  solvers just are better at solving the complex valued systems than those using real arithmetic, in terms of number of iterations  or at times even in success in solving.Nice weekend everybody!

ettaka commented 1 year ago

Amazing, good job! You are the man Juha!

Have a great weekend!

juharu commented 1 year ago

Linear System ILU Order = 0 Linear System Convergence Tolerance = 1.e-6

I'd recommend still adding the former of the above lines (not much help otherwise) + you can then change the convergene tolerance stricter, like in the latter line, and still have results in reasonable number of iterations (~100 iterations or so).

arvedes commented 1 year ago

Thanks a lot for your comments! I changed the settings for case_circuits.sif, see #14. We've now got

@juharu I don't know what's the reason for the case_layer-conductivity to fail with mpi. I tried the test cases for the impedance BC with mpi, works well. Would it make sense to try different solver settings?

juharu commented 1 year ago

Hi Arved

Thanks for all the good work!

Would it make sense to try different solver settings? I don't think so, to me it seems that this just isn't implemented in parallel (I might be wrong of course). Some of us should have a look, I guess.

Br, Juha

juharu commented 1 year ago

I just commited to github "devel" a few small fixes, that seemingly allow the "case_layer_conductivity.sif" pass also in parallel. In addition replacing "Linear System Preconditioning = none" with "Linear System Preconditioning = ILU" in the "MGDynamics" solver will speed things up considerably both serially and in parallel.

juharu commented 1 year ago

In addition replacing "Linear System Preconditioning = none" with "Linear System Preconditioning = ILU" in the "MGDynamics" solver will speed things up considerably both serially and in parallel.

Also "case_coil-solver.sif" speeds up considerably with the above change. As propably does "case_scaled-conductivity.sif", didn't try it though.

arvedes commented 1 year ago

Thanks a lot, Juha! I just compiled & tried again, works well also on my computer :)