Edges of GPU parallelization influence mesh and phasespace dumps

[ervw12]

The edges of the GPU parallelization are clearly visible on the field and phasespace openPMD dumps here: gpuedge.pdf

The edges seem to stretch the Bx component in my co-moving setup of an proton beam propagating in plasma. But all other mesh dumps show a similar stretching at those edges. Further do the plasma electrons gain a higher parallel momentum at the edges seen in the x_parallel p_parallel phasespace.

[psychocoderHPC]

Could you please explain what you simulated and what we can see in the output? @sbastrakov and I tried to understand why you think there is something wrong. We can see that on the right plot there are some unregulated patterns.

[sbastrakov]

Please also attach your .param and .cfg files, maybe it would also help us to understand and try to reproduce

[sbastrakov]

Theoretically (if everything works as intended), the phase space output should have no GPU edge effects. Same for the general grid output.

[ervw12]

Again, thank you very much for that fast response!

I am simulating a proton beam in a co-moving frame, propagating in a cylindrically shaped plasma to show current filamentation. For that I am looking on the ypx phasespace of the plasma electrons, which should get accelerated to introduce an velocity anisotropy needed for the Weibel instability. They actually get accelerated but there should not be any beat effect visible (those small stripes, where some electrons have significantly higher momentum). Of course I cannot say that these stripes are exactly at the edges of the parallelization but those stripes are equidistant, which already seems odd.

I have not always seen those irregular patterns but especially for the bigger runs they occurred. I am using openPMD with ADIOS2.

Here my .param and .cfg files: gpuedge.zip

[sbastrakov]

Thanks. You are right, when such "oddly evenly spaced" effects happen, one has to suspect that something goes wrong with domain decomposition and data exchange.

[PrometheusPi]

According to your param file, you use either full density 7e20 m^-3 or zero density for your electron distribution, depending on the radial position.

dens = (math::sqrt(math::pow(x1,2.0)+math::pow(x3,2.0))<0.0020085384592000485?1.0:0.0);

If you would run through a homogeneous plasma, the sin depth would be around 200µm. But with this density setup, the front of your proton bunch will push aside the electrons within <<200µm and then the rest of the bunch interacts with a much lower density. Thus your wake structure might cause periodic structures at much longer length. To cross check this, could you please run a test case with a homogeneous density across all radii?

How many GPUs are you using in longitudinal (y) direction?

[sbastrakov]

In the cfg file in at attached zip, there are 1x20x1 gpus

[ervw12]

Sorry, I have not had the time to start another bigger simulation yet. And somehow with my smaller simulations I have not encountered that problem.

@PrometheusPi but if I do set a homogenous density across all radii, would it not be necessary to set periodic boundaries? I once tried to set a homogenous background density with PML, which resulted in very strange behaviour of the plasma at the boundary (I am using the --e_boundaryOffset flag).

[PrometheusPi]

@ervw12 not necessarily - we commonly use absorbing or PML boundaries with (transversely) homogeneous densities. As long as the laser is not too strong an the borders, this is a well tested and working setup.

What do you mean with:

I once tried to set a homogeneous background density with PML, which resulted in very strange behavior of the plasma at the boundary (I am using the --e_boundaryOffset flag).

[sbastrakov]

@ervw12 could you attach your setup with this "weird" behavior ? I think it was also mentioned before by someone, but without details. I have an idea of what we may be doing weirdly there, would like to try it.

[PrometheusPi]

@ervw12 what is the current status of this issue - could you contact @sbastrakov by other means (mail etc.)

[ervw12]

@PrometheusPi sorry for not answering. I could not yet replicate the error with my smaller simulations I am currently running. I have contacted @sbastrakov by mail (hopefully I found the right one, pls let me know if not) and will contact him again if the problem occurs again.