Asymmetry issue in two particle test with nodal PSATD (with no single periodic box)

oshapoval commented 2 years ago

Two particles test with nodal PSATD and no periodic single box (psatd.periodic_single_box_fft = 0) demonstrates slight asymmetry. The input script and plots of Ez field and its normalized difference Ez_diff = np.abs(Ez[:,:] - Ez[:,::-1])/np.max(Ez) are attached below.

Tried @RemiLehe suggestions:

to run initial script with heavier particles (i.e., ion.mass = 1e3 and electron.mass = 1e3) = > no change in the amplitude of Ez_diff;
rerunning with commented out all FillBoundary in the Source/Evolve/WarpXEvolve.cpp => no changes in the amplitude of Ez_diff;

max_step = 50
amr.n_cell = 128 128
amr.max_level = 0
#amr.blocking_factor = 128
#amr.max_grid_size = 64
warpx.numprocs = 1 1

#warpx.safe_guard_cells = 1
psatd.periodic_single_box_fft = 0

warpx.do_nodal = 1

geometry.coord_sys   = 0

geometry.prob_lo     = -24.7552 -24.7552
geometry.prob_hi     =  24.7552 24.7552

psatd.nox = 16
psatd.noy = 16
psatd.noz = 16

psatd.nx_guard = 8
psatd.ny_guard = 8
psatd.nz_guard = 8

geometry.dims = 2
# Boundary condition
boundary.field_lo = periodic periodic
boundary.field_hi = periodic periodic

particles.use_fdtd_nci_corr = 0
algo.current_deposition = direct
algo.charge_deposition = standard
algo.field_gathering = momentum-conserving
algo.particle_pusher = vay
algo.maxwell_solver = psatd

#################################
############ NUMERICS ###########
################################
warpx.serialize_ics = 1
warpx.verbose = 1
warpx.cfl = 1.

# Order of particle shape factors
algo.particle_shape = 3

particles.nspecies = 2
particles.species_names = electron ion

electron.charge = -q_e
electron.mass = m_e
electron.injection_style = "SingleParticle"
electron.single_particle_pos = 0.0 0.0 0.0
electron.single_particle_vel = 0.0 0.0 0.5
electron.single_particle_weight = 1.

ion.charge = q_e
ion.mass =  m_p
ion.injection_style = "SingleParticle"
ion.single_particle_pos = 0.0 0.0 0.0
ion.single_particle_vel = 0.0 0.0 -0.5
ion.single_particle_weight = 1.

# Diagnostics
diagnostics.diags_names = diag1
diag1.intervals = 1
diag1.diag_type = Full
diag1.format = openpmd
diag1.species =  electron ion
diag1.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz  rho rho_ion rho_electron
diag1.write_species = 1

oshapoval commented 2 years ago

Here is the comparison between nodal and hybrid PSATD for the same two particle test with domain decomposition along x or z (2 grids of size 64x128 or 128x64, respectively). Only hybrid with domain decomposition along z does not seem to have asymmetry issues in this case.

RemiLehe commented 2 years ago

I found an even simpler case in which the problem shows up: the emission of a laser by an antenna: Screen Shot 2022-02-11 at 10 57 18 AM Here is the input script: inputs.txt

RemiLehe commented 2 years ago

Here is the plot of E[:, :] - E[:, ::-1] at iteration 100 Screen Shot 2022-02-11 at 11 00 05 AM

EZoni commented 2 years ago

As discussed on Slack, this might be related to the issue that we tried to address with #1839 in the past. If confirmed, I can try to open a new PR to implement that change from fully nodal to fully cell-centered.

To be precise, this is the type of operations that most likely break the symmetry in the nodal case: https://github.com/ECP-WarpX/WarpX/blob/ca1b886ecb26c8bd20018c4ecfd71a70f11c1d3f/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp#L247

oshapoval commented 2 years ago

Tried @EZoni fix on 2 particles test. And it fixes the issue with nodal PSATD even with domain decomposition along x and z.

ECP-WarpX / WarpX

Asymmetry issue in two particle test with nodal PSATD (with no single periodic box) #2830