Backtransformed diagnostics with mesh refinement simulation

[prkkumar]

Backtransformed diagnostics with mesh refinement simulations segfaults with the following Backtrace:

=== If no file names and line numbers are shown below, one can run
            addr2line -Cpfie my_exefile my_line_address
    to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
    Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
    One can use
            readelf -wl my_exefile | grep my_line_address'
    to find out the offset for that line.

 0: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x50e1e5) [0x55ab69d631e5]
    ?? ??:0

 1: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x50ff8d) [0x55ab69d64f8d]
    ?? ??:0

 2: /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fe2cea57210]

 3: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7fe2cea5718b]

 4: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7fe2cea36859]

 5: /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e911) [0x7fe2cee0b911]
    ?? ??:0

 6: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa38c) [0x7fe2cee1738c]
    ?? ??:0

 7: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3f7) [0x7fe2cee173f7]
    ?? ??:0

 8: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa6a9) [0x7fe2cee176a9]
    ?? ??:0

 9: /lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt19__throw_logic_errorPKc+0x41) [0x7fe2cee0e21e]
    ?? ??:0

10: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x100f2b) [0x55ab69955f2b]

11: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x10b991) [0x55ab69960991]
    ?? ??:0

12: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x175585) [0x55ab699ca585]
    ?? ??:0

13: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x1419df) [0x55ab699969df]
    ?? ??:0

14: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0xd2f90) [0x55ab69927f90]
    ?? ??:0

15: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0xdd390) [0x55ab69932390]
    ?? ??:0

16: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x1fa31e) [0x55ab69a4f31e]
    ?? ??:0

17: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x78232) [0x55ab698cd232]
    ?? ??:0

18: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fe2cea380b3]

19: ./warpx.3d.MPI.OMP.DP.OPMD.PSATD.QED(+0x86e5e) [0x55ab698dbe5e]
    ?? ??:0

===== TinyProfilers ======
main()
WarpX::Evolve()
WarpX::Evolve::step
Diagnostics::FilterComputePackFlush()
FlushFormatOpenPMD::WriteToFile()
WarpXOpenPMDPlot::WriteOpenPMDFields()

[prkkumar]

Error can be reproduced locally with the following input file

##################################
############# PSATD ##############
##################################
algo.maxwell_solver = psatd
warpx.do_nodal = 1
psatd.use_default_v_galilean = 1
psatd.nox = 12
psatd.noy = 12
psatd.noz = 12
psatd.nx_guard = 12
psatd.ny_guard = 12
psatd.nz_guard = 12
particles.use_fdtd_nci_corr = 0
algo.current_deposition = direct
warpx.do_single_precision_comms = 0

amr.blocking_factor = 64
geometry.dims = 3

#################################
########## MESH PATCH ###########
#################################
amr.max_level = 1

#################################
######### BOX PARAMETERS ########
#################################
my_constants.nx = 32 #128/2
my_constants.nz = 32 #3072/16
my_constants.xmax = 50e-6
my_constants.zmin = -150e-6
my_constants.zmax = 1e-6
warpx.zmax_plasma_to_compute_max_step = 5.e-6
amr.n_cell = nx nx nz
boundary.field_lo = periodic periodic damped
boundary.field_hi = periodic periodic damped
#physical domain when running in the lab frame
geometry.prob_lo = -xmax -xmax zmin
geometry.prob_hi = xmax   xmax zmax
warpx.fine_tag_lo = -5.e-6  -5.e-6  -100.e-6
warpx.fine_tag_hi =  5.e-6   5.e-6  -60.e-6
warpx.refine_plasma = 0

#################################
########## DIAGNOSTICS ##########
#################################
diagnostics.diags_names = diag diag_lab
diag.intervals = 5
diag.format = openpmd
diag.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho
diag.diag_type = Full

diag_lab.diag_type = BackTransformed
diag_lab.do_back_transformed_fields = 1
diag_lab.num_snapshots_lab = 2
diag_lab.dz_snapshots_lab = 1.e-6
diag_lab.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho
diag_lab.format = openpmd
diag_lab.diag_lo = -xmax -2*xmax/nx zmin
diag_lab.diag_hi = xmax   2*xmax/nx   0

#################################
############ NUMERICS ###########
#################################
warpx.verbose = 1
algo.charge_deposition = standard

psatd.do_time_averaging  = 0 #unused -- related to large time step algorith

algo.particle_pusher = vay
algo.particle_shape = 3
warpx.use_filter = 1
warpx.filter_npass_each_dir = 1 1 1
warpx.cfl = 1.

#################################
####### Moving Window ########
#################################
warpx.do_moving_window = 1
warpx.moving_window_dir = z
warpx.moving_window_v = 1.0

#################################
####### BOOST PARAMETERS ########
#################################
warpx.gamma_boost = 30.0
warpx.boost_direction = z

#################################
############ PLASMA #############
#################################
particles.species_names = ions1 electrons1 beam
particles.rigid_injected_species = beam

electrons1.charge = -q_e
electrons1.mass = m_e
electrons1.injection_style = NUniformPerCell
electrons1.num_particles_per_cell_each_dim = 1 1 1
electrons1.momentum_distribution_type = "gaussian"
electrons1.xmin = -46.e-6
electrons1.xmax =  46.e-6
electrons1.ymin = -46.e-6
electrons1.ymax =  46.e-6
electrons1.zmin = 0.0
electrons1.zmax = 1.
electrons1.profile                 = "predefined"
electrons1.predefined_profile_name = "parabolic_channel"
#predefined_profile_params = z_start   ramp_up   plateau   ramp_down   rc       n0
electrons1.predefined_profile_params = 0.0 1.e-9 .05 1.e-9 20.e-6 3.4e23
electrons1.do_continuous_injection = 1
ions1.charge = q_e
ions1.mass = 1.6726219236900001e-29
ions1.injection_style = NUniformPerCell
ions1.num_particles_per_cell_each_dim = 1 1 1
ions1.momentum_distribution_type = "gaussian"
ions1.xmin = -46.e-6
ions1.xmax =  46.e-6
ions1.ymin = -46.e-6
ions1.ymax =  46.e-6
ions1.zmin = 0.0
ions1.zmax = 1.
ions1.profile                 = "predefined"
ions1.predefined_profile_name = "parabolic_channel"
#predefined_profile_params = z_start   ramp_up   plateau   ramp_down   rc       n0
ions1.predefined_profile_params = 0.0 1.e-9 .05 1.e-9 20.e-6 3.4e23
ions1.do_continuous_injection = 1

beam.charge = -q_e
beam.mass = m_e
beam.injection_style = "gaussian_beam"
beam.x_rms = 2.0e-6
beam.y_rms = 2.0e-6
beam.z_rms = 5.0e-6
beam.x_m = 0.
beam.y_m = 0.
beam.z_m = -80.e-6
beam.npart = 50000
beam.q_tot = -3.5e-10
beam.momentum_distribution_type = "gaussian"
beam.ux_m = 0.
beam.uy_m = 0.
beam.uz_m = 1.e9
beam.ux_th = 0.
beam.uy_th = 0.
beam.uz_th = 0.
beam.zinject_plane = 0.0
beam.rigid_advance = true

#################################
############# LASER #############
#################################
lasers.names        = laser1
laser1.profile      = Gaussian
laser1.position = 0. 0. -1.e-9  # This point is on the laser plane
laser1.direction    = 0. 0. 1.      # The plane normal direction
laser1.polarization = 0. 1. 0.      # The main polarization vector
laser1.e_max        = 6.82274e12       # Maximum amplitude of the laser field (in V/m)
laser1.profile_waist = 20.e-6       # The waist of the laser (in meters)
laser1.profile_duration = 7.33841e-14   # The duration of the laser (in seconds)
laser1.profile_t_peak = 1.46764864e-13 # The time at which the laser reaches its peak (in seconds)
laser1.profile_focal_distance = 0.
laser1.wavelength = 0.8e-6         # The wavelength of the laser (in meters)
laser1.do_continuous_injection = 1

[RevathiJambunathan]

PRs #2967 and #2986 have been merged in development that could help with this issue. @prkkumar could you try these cases with the most recent WarpX development ?

[RevathiJambunathan]

Thank you @prkkumar for testing. Prabhat mentioned offline that this is still crashing for plotfiles. I am looking into it

Update : I tested the input here for a case without mesh-refinement and removed diag_lo, and diag_hi for the BTD and it segfaults with

amrex::Abort::0::ParticleContainer::locateParticle(): invalid particle. !!!
SIGABRT

Backtrace.0.txt