Homogenous density implementation erroneous?

[steindev]

When initializing an electron density with the Homogenous profile, the particle density at the boundary is lower than the density in the rest of the simulation box.

This is easy to see when running the 1 gpu case of thermal test and analyzing the openPMD output.

(transverse dimensions cut to half of the box size for better visibility in the image)

Updated 1.cfg and jupyter analysis notebook in (https://cloud.hzdr.de/s/J5JfdAjm3Xiab9P). Answer 1234 when asked while accessing.

@sbastrakov could you please investigate or assign someone else?

[psychocoderHPC]

Please keep in mind that the density is taken the particle shape into account. Therefore diffetent densities at the bondery are expected. The strange thing is that the left and right, bottom and upper side look diffetent.

[psychocoderHPC]

Which shape do you use TSC or PQS?

[sbastrakov]

What @psychocoderHPC wrote is partly correct. We actually treat it differently in different places: some use shape, some don't.

[psychocoderHPC]

@sbastrakov all derived fields uses the shape, you can define a derived field with a counter shape to avoid the particle shape.

[sbastrakov]

This is true. However it doesn't contradict what I wrote. E.g. when generating macroparticles we don't account for shapes, only generate centers and then slap whatever shape it is on top.

[psychocoderHPC]

@steindev do you enabled periodic bounderies?

[sbastrakov]

It is not in the .cfg file, so I assume no. Edit: but the image is from iteration 0, so boundaries should not fully explain it.

[psychocoderHPC]

But even without periodic I am wondering why it is not symetric. Do we have an offset error in your calculation?

[sbastrakov]

It is only half (by each side, so a quater) of the plot. The grid is of size 128

[psychocoderHPC]

Ahh if the shape is TSC and the other side looks equal than all should be fine.

[sbastrakov]

Perhaps what we should consider doing, is when outputting cumulative fields (like densities), do a reduction along periodic boundaries. Like we do internally with current densities used for calculations. As those would be actual "physical" values for near-boundary cells, and what we output currently aren't. So we now rely on a user understanding it, and doing it manually when necessary.

Nvm, this is wrong. We are already doing what i "suggested"

[psychocoderHPC]

Maybe I understand something of your comment wrong but if the user is activating periodic bounderies the data will look correct. Data will be synchronized (merged) if periodic is enabled.

What do you mean by reduction along periodic boundaries?

[sbastrakov]

Ah, you are right. I just thought of smth else and didn't realize we are already doing what I wrote.

[sbastrakov]

So as a short summary @steindev . When you are running without periodic boundaries, the near-boundary density values (and all fields that are sums of particles' contributions) are expected to differ from the rest due to form-factors. What happens is that macroparticles with centers in such cells will contribute parts of their "density" to the guard area, while getting no contributions from there due to no macroparticles in that area. Cells in the middle will contribute to others and get back on average the same. If you run with periodic boundaries, all cells become "in the middle" and that should be no problem, because the particles from "the other side" will contribute there.

[steindev]

Ok, this would explain a lower density at all boundaries. But I see it only at one side. The full domain looks like

and only the upper right quadrant

ThermalTest uses the standard particle shape TSC.

[sbastrakov]

Could it be that it is due to the grid positions where we evaluate density being at origins of cells? Then the grid layer symmetrical to the leftmost would be 128, which is not part of the output. And the layer 127 we see is 1 cell distance from the "right-side" boundaries. With TSC the layer at 127 should be mostly correct, the higher the shape the most should be the divergence from the expected value

[steindev]

Could it be that it is due to the grid positions where we evaluate density being at origins of cells?

Sound reasonable. Will investigate with highest shape and see if it makes the expected difference.

[psychocoderHPC]

Could it be that it is due to the grid positions where we evaluate density being at origins of cells? Then the grid layer symmetrical to the leftmost would be 128, which is not part of the output. And the layer 127 we see is 1 cell distance from the "right-side" boundaries. With TSC the layer at 127 should be mostly correct, the higher the shape the most should be the divergence from the expected value

Yes this could be. Take care where the density is defined. Density is defined at the origin of a cell, this means at position 0,0,0.

[steindev]

So with @sbastrakov's explanation I would have expected that I see such a "layer" of lower density on all boundaries if I use a higher particle shape. Is this expectation correct? Because I don't see a layer on the other boundaries when using PCS.

[sbastrakov]

It would be much less "layerish" I think than what you see on the left-most output. Roughtly same as output at x == 1 or y == 1

[sbastrakov]

(as non-existing 128 is symmetric to 0, and 127 symmetric to 1)

[sbastrakov]

I think when looking for differences, it would be easier if the expected value is subtracted. Then the exact value would be 0 and discrepancies from it easier seen in color.

[psychocoderHPC]

So with @sbastrakov's explanation I would have expected that I see such a "layer" of lower density on all boundaries if I use a higher particle shape. Is this expectation correct? Because I don't see a layer on the other boundaries when using PCS.

For TSC the lower values are in cell 128 not 127. Cell 128 is not part of the output because it is in the guard area.

[steindev]

To sum up: @sbastrakov is correct. The lower density on the "left-hand" side of the domain is due to the calculation of particle density at grid nodes (and not in cell centers). Due to that left-hand neighbors for the left hand boundaries are missing.

This is confirmed by running the same simulation with periodic boundaries where left hand neighbors are present by periodicity. See the following plots showing relative deviation of density output from expected value

for absorbing boundary conditions

and periodic boundary conditions:

(black is where the deviation from the expected value is largest.)