possible png-plugin density-value problem

[PrometheusPi]

This issue was generated during a simulation by @finnolec on taurus/ZIH/TU Dresden.

If you take the png output, it appears that there are regions with lower than expected density.

(see red arrows, iteration 23000).

If one compares this to the output of the openPMD-api plugin (adios2), such significantly reduced density can not be observed.

(iteration 23395)

However, if we zoom in the strange spatial regions, one can see a slightly higher density fluctuation:

which is better detectable if one plots the electric field E_y in logarithmic scale:

The strange physics might originate from an error in how the sideways laser is implemented and is not topic of this issue. The question is why we see such strongly reduced density values in the png output if by both what we observe in the openPMD-api output and what we expect from the physics behavior, the density should not be reduced.

[sbastrakov]

Could it be an unfortunate slice in png? E.g. if there is some antisymmetry around the center, and the slice at center was used? The png plugin seems to select particles based on the cell they are in, without taking into account form factors like the density evaluation does.

[sbastrakov]

However, from the regular pattern one may also suspect the gathering of results between processes.

[PrometheusPi]

@sbastrakov Does this mean, that if I want to reproduce the png-plugin's approach, I would need to look at the particle/electron positions on the sliced/center cell directly using a NGP (Hockney and Eastwood naming) particle shape?

[psychocoderHPC]

@sbastrakov Does this mean, that if I want to reproduce the png-plugin's approach, I would need to look at the particle/electron positions on the sliced/center cell directly using a NGP (Hockney and Eastwood naming) particle shape?

I think to reproduce what png's are doing you should use the Counter shape. In the png plugin, we add all densities to the cell where the particle is located.

[psychocoderHPC]

@PrometheusPi How many GPUs @finnolec is using. I am wondering if the not-so-dense areas are full GPUs.

[sbastrakov]

I am not sure tbh. I believe NGP should be similar, but not identical - it should contribute to the closest grid node but notnecessarily to the current cell (however that depends on the implementation details of the plugin which i am not familiar with). Counter shape should in principle be that, but it is generally not safe to use (as it is not a real shape in our sense, but a stub that works in some situation).

[PrometheusPi]

@psychocoderHPC No, the not-so-dense regions are coming from the sliding window and are unrelated to the GPU distribution.

There seems to be no particle data in the openPMD-api output. @finnolec is this correct or am I just too stupid to read the output? If there is no particle data, there is no way to cross check @sbastrakov hypothesis regarding NGP.

[finnolec]

@PrometheusPi How many GPUs @finnolec is using. I am wondering if the not-so-dense areas are full GPUs.

My cfg setup is:

TBG_devices_x=2
TBG_devices_y=16
TBG_devices_z=4

So I'm using 128 GPUs. I think the longitudinal extent (x in plot, y in picongpu) of the weird rectangles is the length of 1/15th of the simulation box, so this is somehow related to the GPU distribution. But the transverse extent (y direction in plot, z direction in picongpu) is not related to the GPUs... Iirc it was related to the length of the second, low intensity laser I added to the simulation with #3233.

There seems to be no particle data in the openPMD-api output. @finnolec is this correct or am I just too stupid to read the output? If there is no particle data, there is no way to cross check @sbastrakov hypothesis regarding NGP.

Yes, I didn't disabled particle data output in openPMD for performance reasons (and I didn't need it)...

[PrometheusPi]

@finnolec would it be possible to rerun one of the affected simulations but with particle output enabled?

[finnolec]

@finnolec would it be possible to rerun one of the affected simulations but with particle output enabled?

I just restarted the simulation. I will send you the output path as soon as there is output

[PrometheusPi]

Thank you @finnolec.

[finnolec]

I tried to run the simulations with field + particle output but the simulation crashed because the buffer size for my openPMD output was too small... Currently I don't really have time to rescale everything again so that the simulation runs again

[PrometheusPi]

@finnolec Thanks for the feedback. As discussed offline, there is no time issue. It just would be nice to validate our current presumption.