search

ECP-WarpX / WarpX

WarpX is an advanced electromagnetic & electrostatic Particle-In-Cell code.
https://ecp-warpx.github.io
Other
310 stars 196 forks source link

Problems with injection from external file #3185

Open MartaGalbi opened 2 years ago

MartaGalbi commented 2 years ago

Hi all! I am trying to initialize particles from an external file to get a custom distribution with aggregates of particles. When I increase the number of particles in the openPMD file above a certain level, the simulation crashes with the error out of GPU memory. For example, it works with an openPMD file of 7,6GB, but it crashes with one of 8.8GB. However, a simulation with the same resources but using random injection without an external file can easily load the same number of particles. Increasing resources does not help, so I suppose there are some limits due to the loading from the file. Am I right? Is this problem solvable? I am working on Marconi100 with the latest version of WarpX. Input file: input0.txt Job: job.txt Output files: Backtrace.0.txt job.err.txt output.txt

MartaGalbi commented 1 year ago

This problem still exists with the latest version of Warpx. I investigated if a PICMI input could help in initializing such arbitrary densities, but I didn't find a feasible solution. Do you have any suggestions?

ax3l commented 1 year ago

Thank you for the detailed report.

The issue appears because the external file read logic for particles is not yet MPI-parallelized: https://github.com/ECP-WarpX/WarpX/blob/23.06/Source/Particles/PhysicalParticleContainer.cpp#L576-L593 I think the reason for that is that people used this function so far only to load particle beams (usually a few 10k particles) and not full targets/background plasma distributions.

The way to update this is:

We do similar logic in regular initialization of plasma profiles.

When we add particles, we automatically redistribute them on the correct rank as long as the position is valid in the simulation box (otherwise they get removed).

Do you like to contribute this logic to the linked location or do you like to wait until someone has time to give it a try on our end? :)

MartaGalbi commented 1 year ago

Hi @ax3l, Thank you for answering my issue, I want to try to contribute following your indications. However, if I see that I have many difficulties in doing it, I will wait for someone more expert to solve it. I will update you on my attempt.

PhysicsDan commented 1 year ago

Hi @MartaGalbi, I was just wondering have you had any success with distributing the particle initialisation? Thanks, Daniel