search

ECP-WarpX / WarpX

WarpX is an advanced electromagnetic & electrostatic Particle-In-Cell code.
https://ecp-warpx.github.io
Other
310 stars 196 forks source link

Segmentation fault when running 2D uniform plasma simulation with implicit PIC #4671

Closed oshapoval closed 9 months ago

oshapoval commented 9 months ago

I would like to report an issue I am observing when using implicit PIC. Specifically, when running a 2D uniform plasma script (please find attached it below) with implicit PIC and time step dt = 0.1/wpe I am having the following segfault:

Level 0: dt = 1.772590711e-18 ; dx = 2.07581768e-10 ; dz = 2.07581768e-10

Grids Summary:
  Level 0   1 grids  65536 cells  100 % of domain

-------------------------------------------------------------------------------
--------------------------- MAIN EM PIC PARAMETERS ----------------------------
-------------------------------------------------------------------------------
Precision:            | DOUBLE
Particle precision:   | DOUBLE
Geometry:             | 2D (XZ)
Operation mode:       | Electromagnetic
                      | - vacuum
-------------------------------------------------------------------------------
Current Deposition:   | Esirkepov 
Particle Pusher:      | Boris 
Charge Deposition:    | standard 
Field Gathering:      | energy-conserving 
Particle Shape Factor:| 1
-------------------------------------------------------------------------------
Maxwell Solver:       | Yee 
-------------------------------------------------------------------------------
For full input parameters, see the file: warpx_used_inputs

--- INFO    : Writing openPMD file diags/diag1000000
STEP 1 starts ...
amrex::Abort::0:: (258,89,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
SIGABRT
amrex::Abort::0:: (258,33,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
amrex::Abort::0:: (258,21,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
amrex::Abort::0:: (258,74,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
amrex::Abort::amrex::Abort::amrex::Abort::0:: (258,129,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
amrex::Abort::amrex::Abort::0:: (258,143,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
0:: (258,106,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
amrex::Abort::0:: (258,8,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
0:: (258,99,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
0:: (258,44,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
amrex::Abort::amrex::Abort::0:: (258,169,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
0:: (258,167,0,0) is out of bound (-2:257,-2:258,0:0,0:0) !!!
See Backtrace.0.11 file for details
Abort(6) on node 0 (rank 0 in comm 496): application called MPI_Abort(comm=0x84000000, 6) - process 0

I am also attaching the corresponding E_z snapshots at the first time step ran with both implicit solver and multi-J PSATD for comparison. To set multi-J PSATD I used:


# psatd
algo.current_deposition = direct
algo.maxwell_solver = psatd
algo.particle_pusher = vay
algo.particle_shape = 3
warpx.grid_type = collocated
warpx.use_filter = 1

# Multi-J
psatd.nox = 16
psatd.noy = 16
psatd.noz = 16

psatd.J_in_time=linear
psatd.rho_in_time=linear
warpx.do_dive_cleaning=1
warpx.do_divb_cleaning=1
warpx.do_multi_J=1
warpx.do_multi_J_n_depositions=2
algo.field_gathering = energy-conserving

@dpgrote @RemiLehe Could you please double-check if anything missing in the set up of the implicit solver in my input? Do I need to increase algo.max_picard_iterations?

inputs_2d.txt

Screenshot 2024-02-05 at 6 29 37 AM
EyaDammak commented 9 months ago

@RemiLehe, it seems like an issue that I had several times (#4593).

JustinRayAngus commented 9 months ago

Two comments.

1) you have algo.particle_shape set twice in your input. The code is using the last instance, which is 1. I would recommend using 2 or greater for implicit.

2) It looks like the grid is much more refined than what is normally used. My guess is that your segfault is because you are strongly violating CFL with a grid that that is 6.4 times more refined than the default. Try lowering your time step by 6.4 as well.

oshapoval commented 9 months ago

Thank you @JustinRayAngus for your comments. Taking much smaller dt, for example, cdt/dz=0.4 resolves the issue. With cdt/dz=0.99 the Picard implicit solver still fails to converge (expected with the current implementation, as discussed with @JustinRayAngus during the meeting).

Also, setting non-zero tolerance, e.g. algo.picard_iteration_tolerance = 1e-8 gives an explicit error message in this case:

Level 0: dt = 6.897240119e-19 ; dx = 2.07581768e-10 ; dz = 2.07581768e-10

Grids Summary:
  Level 0   4 grids  65536 cells  100 % of domain
            smallest grid: 128 x 128  biggest grid: 128 x 128

-------------------------------------------------------------------------------
--------------------------- MAIN EM PIC PARAMETERS ----------------------------
-------------------------------------------------------------------------------
Precision:            | DOUBLE
Particle precision:   | DOUBLE
Geometry:             | 2D (XZ)
Operation mode:       | Electromagnetic
                      | - vacuum
-------------------------------------------------------------------------------
Current Deposition:   | Esirkepov 
Particle Pusher:      | Boris 
Charge Deposition:    | standard 
Field Gathering:      | energy-conserving 
Particle Shape Factor:| 2
-------------------------------------------------------------------------------
Maxwell Solver:       | Yee 
-------------------------------------------------------------------------------
For full input parameters, see the file: warpx_used_inputs

--- INFO    : Writing openPMD file diags/diag1000000
STEP 1 starts ...
Max delta 1 1 1
Max delta 2 1.213909068 2.643546784
Max delta 3 0.9227582959 1.112906664
Max delta 4 1.425462081 1.936705744
Max delta 5 1.243993083 1.37485431
Max delta 6 1.673315978 1.717988368
Max delta 7 1.440593203 1.531724343
Max delta 8 1.602452699 1.610555808
Max delta 9 1.529315659 1.557272897
Max delta 10 1.566588062 1.540916475
Max delta 11 1.54492169 1.537869772
Max delta 12 1.544086364 1.538679575
Max delta 13 1.540435062 1.537356466
Max delta 14 1.531621212 1.524474031
Max delta 15 1.522751706 1.523423567
Max delta 16 1.523123923 1.522527369
Max delta 17 1.521771538 1.521650509
Max delta 18 1.521342712 1.520866294
Max delta 19 1.520477015 1.520133148
Max delta 20 1.519894535 1.5194663
Max delta 21 1.519225727 1.518850132
Max delta 22 1.518663821 1.518284754
Max delta 23 1.518107228 1.517762341
Max delta 24 1.517605924 1.517279676
Max delta 25 1.517130319 1.516831759
Max delta 26 1.516691833 1.516415107
Max delta 27 1.516280032 1.51602598
Max delta 28 1.51590822 1.515661528
Max delta 29 1.515577135 1.515319386
Max delta 30 1.515260194 1.515077093
Max delta 31 1.514956817 1.514917832
Picard iterations = 31, Eerror = 1.514956817, Berror = 1.514917832
amrex::Abort::0::
### ERROR   : The Picard implicit solver failed to converge after 31 iterations,
#            with Eerror = 1.51496, Berror = 1.51492 with a tolerance of 1e-08
#            (/Users/oshapoval/workspace/WarpX/Source/Evolve/WarpXOneStepImplicitPicard.cpp:245)
 !!!
SIGABRT