mccoys
commented
2 years ago Hi What crash do you see without vectorization? And which version of smilei?
Closed StephenDiIorio closed 2 years ago
mccoys
commented
2 years ago Hi What crash do you see without vectorization? And which version of smilei?
StephenDiIorio
commented
2 years ago I believe it was the same error whether or not I removed the Vectorization block from the input deck. And the smilei version was 4.6-351-g4230c5878-master.
StephenDiIorio
commented
2 years ago Yea it was the same error. I've attached it anyway in case it is at all useful.
mccoys
commented
2 years ago Do you reproduce the error on any other case? For instance, have you tried some of the benchmarks?
StephenDiIorio
commented
2 years ago I ran the stopping power benchmarks and they ran with no issue.
mccoys
commented
2 years ago Plot some of your fields at smaller time intervals. You will see your simulation explodes. Either you are not neutral or you dont resolve some characteristic lengths (c/w_p or the debye length)
mccoys
commented
2 years ago Actually I believe it is related to the EM_boundary_conditions_k that is probably incorrect.
StephenDiIorio
commented
2 years ago I see. After playing around with this and commenting out that one line from the input deck, the simulation does complete. But I guess I don't understand what those values represent then. My interpretation from the documentation is that the values you provide for that variable are the unit vectors describing the direction you anticipate the incident k vector to approach the boundary with. With this simulation, I anticipate that the laser comes in from the left wall, hits the solid target (which fills the bottom right triangle of the simulation box) and exits via the top boundary. There would probably be some fields that also propagate through the solid and leave through the bottom and right boundaries. So that leaves me with the question as to what I'm actually describing with EM_boundary_conditions_k?
Separately, I'm trying to analyze the results of the simulation that did finish and am getting funny behavior. Like I mentioned previously, I would have liked a laser to come in from the left wall and interact with a solid target. However, the fields I'm getting make absolutely no sense to me. I've attached them.
Interestingly, if I remove all the species from the input deck and only simulate a laser propagating, I get what you would expect the laser fields to look like. Do you have any suggestions as to why I am seeing what I am?
Thank you for your help with all of this!
mccoys
commented
2 years ago Concerning the parameter EM_boundary_conditions_k, I do not recall exactly the acceptable bounds. The vector should represent the direction of wavevectors that are best absorbed. @beck-llr Do you have more info? The answer might be that those vectors should always be in a direction towards inside the box, but that may not be best rule.
Concerning your results, you clearly have a problem with initial fields (they are not 0, and far from it). The issue is that the Poisson solver finds a non-neutral case overall, which is not good.
First you should have only 1 electron species and 1 ion species (because I believe that fits your situation). The preplasma should be included identically in both populations, and you can do it at once using a single function containing both the preplasma and the solid target density profile. Then, if you wish to start without any ionisation, put 0 density in electrons, and put 0 charge in ions.
beck-llr
commented
2 years ago The optimal wave absorption on a given face will be along the specular reflection of k_inc on the considered face.
That means that, for face xmin for instance, if you have defined a k_inc [x,y,z], the best absorption will be along k_abs [-x,y,z].
Therefore it doesn't make sense to define k_inc going out of the simulation domain since it means wave getting in will be absorbed which is the opposite of what you want when using absorbing boundaries (you want waves getting out of the domain to be absorbed).
There is indeed a problem with the definition of EM_boundary_conditions_k in your case. Unless you know perfectly what you are doing, I strongly suggest to use the default values for this parameter.
Description
I'm trying to run a simulation in smilei but am running into some issues I don't know how to fix. I submit the attached input deck to my cluster, and it runs for about a half-hour before encountering the attached error. I have tried turning off the vectorization portion of my input deck because I assumed that was what the error was referring to, but that does not seem to fix it.
Input Deck
```python import math import numpy as np Main( geometry="2Dcartesian", number_of_patches=[8, 8], interpolation_order=2, timestep=0.01313, simulation_time=121, cell_length=[0.03125, 0.03125], grid_length=[60, 60], EM_boundary_conditions=[["silver-muller"]], EM_boundary_conditions_k=[[1., 0.], [1., 0.], [0., -1.], [0., 1.]], random_seed=0, reference_angular_frequency_SI=4.41e19, print_every=426 ) Vectorization( mode="on", reconfigure_every=20, initial_mode="on" ) def prePlasmaProfile(x1, x2): return 1450. * np.heaviside(x2 - x1, 1.) * np.exp((x1 - x2) / 1.) Species( name="preplasma", position_initialization="random", particles_per_cell=32, momentum_initialization="maxwell-juettner", mean_velocity=[0., 0., 0.], temperature=[0.0014, 0.0014, 0.0014], mass=1, charge=-1, number_density=prePlasmaProfile, boundary_conditions=[["remove"]] ) def molyProfile(x1, x2): return 1450. * np.heaviside(x1 - x2, 1.) Species( name='moly', position_initialization="random", particles_per_cell=32, momentum_initialization="cold", mean_velocity=[0., 0., 0.], temperature=[0., 0., 0.], atomic_number=42, mass=174900, charge=0, number_density=molyProfile, maximum_charge_state=14, boundary_conditions=[["remove"]], ionization_model="tunnel", ionization_electrons="eTarget", ) Species( name="eTarget", position_initialization="random", particles_per_cell=32, momentum_initialization="cold", mean_velocity=[0., 0., 0.], temperature=[0., 0., 0.], mass=1, charge=-1, number_density=0., boundary_conditions=[["remove"]] ) LaserGaussian2D( box_side="xmin", a0=0.4, omega=2 * math.pi, polarization_phi=0, ellipticity=0, incidence_angle=0, focus=[28.7, 30], # not sure if the 30 in y is actually correct waist=1.7, time_envelope=tgaussian(duration=74, fwhm=18.5) ) Collisions( species1=["eTarget"], species2=["moly"], ionizing=True ) DiagFields( every=426, fields=["Ex", "Ey", "Rho", "Rho_moly", "Rho_eTarget"], ) DiagParticleBinning( name="spec_rho", deposited_quantity="weight_charge", every=426, species=["eTarget", "moly"], axes=[ ["x", 0, 60, 1920], ["y", 0, 60, 1920] ] ) DiagParticleBinning( name="charge_distro", deposited_quantity="weight", every=426, species=["eTarget"], axes=[ ["charge", -0.5, 14.5, 15] ] ) DiagParticleBinning( name="gam_pspace", deposited_quantity="weight", every=426, species=["eTarget"], axes=[ ["gamma", 1, 5, 10000] ] ) DiagPerformances( every=213 ) ```Simulation Output
``` _ _ ___ _ | | _ \ \ Version : 4.6-351-g4230c5878-master / __| _ __ (_) | | ___ (_) | | \__ \ | ' \ _ | | / -_) _ | | |___/ |_|_|_| |_| |_| \___| |_| | | /_/ Reading the simulation parameters -------------------------------------------------------------------------------- HDF5 version 1.10.5 Python version 3.7.3 Parsing pyinit.py Parsing 4.6-351-g4230c5878-master Parsing pyprofiles.py Parsing 800nm1.py Parsing pycontrol.py Check for function preprocess() python preprocess function does not exist Calling python _smilei_check Calling python _prepare_checkpoint_dir Calling python _keep_python_running() : [WARNING] Patches distribution: hilbertian [WARNING] simulation_time has been redefined from 121.000000 to 120.992950 to match timestep. [WARNING] Particles cluster width set to : 8 Geometry: 2Dcartesian -------------------------------------------------------------------------------- Interpolation order : 2 Maxwell solver : Yee simulation duration = 120.992950, total number of iterations = 9215 timestep = 0.013130 = 0.594196 x CFL, time resolution = 76.161462 Grid length: 60, 60 Cell length: 0.03125, 0.03125, 0 Number of cells: 1920, 1920 Spatial resolution: 32, 32 Electromagnetic boundary conditions -------------------------------------------------------------------------------- xmin silver-muller, absorbing vector [1, 0] xmax silver-muller, absorbing vector [1, 0] ymin silver-muller, absorbing vector [0, -1] ymax silver-muller, absorbing vector [0, 1] Vectorization: -------------------------------------------------------------------------------- Mode: on Initializing MPI -------------------------------------------------------------------------------- MPI_THREAD_MULTIPLE enabled Number of MPI processes: 36 Number of threads per MPI process : 1 Number of patches: 8 x 8 Number of cells in one patch: 240 x 240 Dynamic load balancing: never Initializing the restart environment -------------------------------------------------------------------------------- Initializing species -------------------------------------------------------------------------------- Creating Species #0: preplasma > Pusher: boris > Density profile: 2D user-defined function (uses numpy) Creating Species #1: moly > Pusher: boris [WARNING] Performances of advanced physical processes which generates new particles could be degraded for the moment! [WARNING] The improvement of their integration in vectorized algorithms is in progress. > Density profile: 2D user-defined function (uses numpy) Creating Species #2: eTarget > Pusher: boris > Density profile: 2D built-in profile `constant` (value: 0.000000) Initializing laser parameters -------------------------------------------------------------------------------- Laser #0: separable profile omega : 6.28319 chirp_profile : 1D built-in profile `tconstant` (0.000000) time envelope : 1D built-in profile `tgaussian` (start: 0.000000, duration: 74.000000, sigma: 123.440594, center: 60.500000, order: 2.000000) space envelope (y) : 1D user-defined function space envelope (z) : 1D user-defined function phase (y) : 1D user-defined function phase (z) : 1D user-defined function delay phase (y) : 0 delay phase (z) : 0 Parameters for collisions #0 : Collisions between species (2) and (1) Coulomb logarithm: 0.000000 Collisional ionization with atomic number 42 towards species `eTarget` Initializing Patches -------------------------------------------------------------------------------- First patch created Approximately 10% of patches created All patches created Creating Diagnostics, antennas, and external fields -------------------------------------------------------------------------------- Created ParticleBinning #0: species eTarget,moly Axis x from 0 to 60 in 1920 steps Axis y from 0 to 60 in 1920 steps Created ParticleBinning #1: species eTarget Axis charge from -0.5 to 14.5 in 15 steps Created ParticleBinning #2: species eTarget Axis gamma from 1 to 5 in 10000 steps Diagnostic Fields #0 : Ex Ey Rho Rho_moly Rho_eTarget Created performances diagnostic finalize MPI -------------------------------------------------------------------------------- Done creating diagnostics, antennas, and external fields Minimum memory consumption (does not include all temporary buffers) -------------------------------------------------------------------------------- Particles: Master 352 MB; Max 352 MB; Global 11 GB Fields: Master 13 MB; Max 13 MB; Global 0.429 GB scalars.txt: Master 0 MB; Max 0 MB; Global 0 GB ParticleBinning0.h5: Master 28 MB; Max 28 MB; Global 0.989 GB ParticleBinning1.h5: Master 0 MB; Max 0 MB; Global 4.02e-06 GB ParticleBinning2.h5: Master 0 MB; Max 0 MB; Global 0.00268 GB Fields0.h5: Master 0 MB; Max 0 MB; Global 0 GB Performances.h5: Master 0 MB; Max 0 MB; Global 0 GB Initial fields setup -------------------------------------------------------------------------------- Solving Poisson at time t = 0 Initializing E field through Poisson solver -------------------------------------------------------------------------------- Poisson solver converged at iteration: 7610, relative err is ctrl = 0.990038 x 1e-14 Poisson equation solved. Maximum err = 0.000000 at i= -1 Time in Poisson : 12.328679 Applying external fields at time t = 0 Applying prescribed fields at time t = 0 Applying antennas at time t = 0 Open files & initialize diagnostics -------------------------------------------------------------------------------- Running diags at time t = 0 -------------------------------------------------------------------------------- Species creation summary -------------------------------------------------------------------------------- Species 0 (preplasma) created with 59013120 particles Species 1 (moly) created with 59013120 particles Species 2 (eTarget) created with 0 particles Expected disk usage (approximate) -------------------------------------------------------------------------------- WARNING: disk usage by non-uniform particles maybe strongly underestimated, especially when particles are created at runtime (ionization, pair generation, etc.) Expected disk usage for diagnostics: File Fields0.h5: 3.02 G File Performances.h5: 309.66 K File scalars.txt: 0 bytes File ParticleBinning0.h5: 618.76 M File ParticleBinning1.h5: 17.79 K File ParticleBinning2.h5: 1.69 M Total disk usage for diagnostics: 3.63 G Keeping or closing the python runtime environment -------------------------------------------------------------------------------- Checking for cleanup() function: python cleanup function does not exist Closing Python Time-Loop started: number of time-steps n_time = 9215 -------------------------------------------------------------------------------- [WARNING] The following `push time` assumes a global number of 36 cores (hyperthreading is unknown) timestep sim time cpu time [s] ( diff [s] ) push time [ns] 426/9215 5.5999e+00 5.3214e+02 ( 5.3214e+02 ) 453 [gl3139:71449:0:71449] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffe02e1e150) ==== backtrace (tid: 71449) ==== 0 0x000000000004f6b5 ucs_debug_print_backtrace() ???:0 1 0x0000000000b58893 SpeciesV::dynamics() /home/diiorios/Smilei/src/Species/SpeciesV.cpp:296 2 0x00000000009a9ef3 VectorPatch::dynamics() /home/diiorios/Smilei/src/Patch/VectorPatch.cpp:357 3 0x0000000000b1cf5b L_main_365__par_region0_2_140() /home/diiorios/Smilei/src/Smilei.cpp:403 4 0x00000000000e7ed3 __kmp_invoke_microtask() ???:0 5 0x00000000000aadf7 __kmp_fork_call() ???:0 6 0x000000000006d020 __kmpc_fork_call() ???:0 7 0x0000000000b20d30 main() /home/diiorios/Smilei/src/Smilei.cpp:365 8 0x0000000000022555 __libc_start_main() ???:0 9 0x00000000005f6804 _start() ???:0 ================================= Stack trace (most recent call last): #9 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in #8 Object "/home/diiorios/Smilei/smilei", at 0x5f6803, in #7 Object "/lib64/libc.so.6", at 0x2b444d52b554, in __libc_start_main #6 Object "/home/diiorios/Smilei/smilei", at 0xb20d2f, in main #5 Object "/sw/arcts/centos7/python3.7-anaconda/2019.07/lib/libiomp5.so", at 0x2b444d18c01f, in __kmpc_fork_call #4 Object "/sw/arcts/centos7/python3.7-anaconda/2019.07/lib/libiomp5.so", at 0x2b444d1c9df6, in __kmp_fork_call #3 Object "/sw/arcts/centos7/python3.7-anaconda/2019.07/lib/libiomp5.so", at 0x2b444d206ed2, in __kmp_invoke_microtask #2 Object "/home/diiorios/Smilei/smilei", at 0xb1cf5a, in main #1 Object "/home/diiorios/Smilei/smilei", at 0x9a9ef2, in VectorPatch::dynamics(Params&, SmileiMPI*, SimWindow*, RadiationTables&, MultiphotonBreitWheelerTables&, double, Timers&, int) #0 Object "/home/diiorios/Smilei/smilei", at 0xb58893, in SpeciesV::dynamics(double, unsigned int, ElectroMagn*, Params&, bool, PartWalls*, Patch*, SmileiMPI*, RadiationTables&, MultiphotonBreitWheelerTables&, std::vectorThis is done with the latest version of smilei and is compiled with the intel compilers. The machine=skylake flag was also used during compilation.
A full list of modules loaded for compilation and runtime are shown here