simulations in a moving window

[cbontoiu]

Hello, Could anyone help me to understand how the moving window works in PIConGPU, please? All I have is the information from the typical grid.param file but it seems that the variable movePoint is no longer used .

So far I have used full size simulations because there was no windowMovePoint in my .cfg files. Is there an example that I can use? I guess there must be more parameters to define the moving window such as its x,y,z dimensions.

My understanding is like this with respect to the figure below: 1) if I enable the moving window from lets say 25% of the y-axis onwards, I will have results in the y = 0,80 nm and then the window will be generated as a cube of some predefined dimensions; 2) are all three cube sides dimensions controllable; I other words could I restrict the transverse simulations sides to something lower than the real simulation domain, when the moving window enters the stage? 3) after the moving window is generated for the y > 80 nm, simulations results will be available only for the region inside the (3D?) moving window; Is this correct? 4) What are the advantages of the moving window. I guess the full simulation domain is replaced by the local volume of the 3D window and this speeds up the simulation. 5) Can we enforce the mesh across the moving window only?

The greatest difficulty is however to get a model started using the moving window. As you can see in the picture below, though I enabled it in the .cfg file, it does not seem to be turned on.

[sbastrakov]

Hello @cbontoiu .

Thanks for your report. We are aware that the moving window parameters are not the most natural, nor well-documented, and plan to improve on it after the upcoming release is finished (hopefully soon) to not break compatibility with the existing setups of our users.

Currently, PIConGPU moving window always slides along Y towards positive direction (so the same direction as lasers), and always with the speed of light. So all the parameters implicitly apply to this direction only.

Your piece of the .cfg file looks reasonable, so the problem must be somewhere else. Perhaps you somehow do not include TBG_movingWindow into the resulting command line of PIConGPU? TBG_movingWindow is merely a variable name and not a keyword or option by itself, and somewhere in the end of your .cfg file all such variables are assembled into command line parameters. In case you are unsure, please provide the full .cfg file.

The movePoint variable is still required in grid.param for now, but for future compatibility please both set this variable and use --windowMovePoint in your .cfg file. In case you did not specify --windowMovePoint in your .cfg file, it will use the movePoint value by default.

Regarding your questions,

1. With the moving window enabled, you start with exactly the same simulated area as would be without it. After the window starts moving at some time step (according to your parameters), it still always simulates the same number of cells and so the same physical volume. However, for different window positions, this simulated volume corresponds to different parts of the coordinate space - it slides towards positive Y with the speed up light (you can also stop the sliding at some time step and continue statically, as usual). The moving window is this simulated area: constant in size, sliding through the coordinate space. Perhaps this picture is of help.
2. As I wrote for question 1, the box size is the same, with or without the moving window, and constant during the simulation, it is just defined by what you set as grid size.
3. Yes, from a user's and output point of view only the cells and macroparticles in the window exist, and are output.
4. I am not familiar enough with your application. As far as I know, generally moving window is mostly used for laser-particle acceleration applications to follow the laser. Then a moving window can be set to start when the laser reaches e.g. 90% of the simulation volume in Y and then starts moving along with the laser. Then we can simulate acceleration length that is longer than the simulation domain in Y, without spending resources on the whole "global domain" according to the linked picture.
5. Yes, from a user's point of view at any moment of time the mesh and all simulation data only exists for the window area. So not only you "can enforce" it, there is no other way.

[PrometheusPi]

@cbontoiu The moving window approach is a method commonly used for LWFA or PWFA simulations that allows to simulate the plasma dynamics behind the laser/beam driver while it propagates for a longer distance than the simulation box. A detailed explaiantion can be found in the PIConGPU wiki.

It is a Galilean frame that moves (usually) with the speed of light along the laser/bean driver. In order to keep the code efficient, PIConGPU not only has a moving simulation box (where physics is simulated) but also slides the entire simulation domain: Imagine you have a simulation using N_x x N_y x N_z cells distributed on G_x x G_y x G_z GPUs in each dimension with G_y >1. This is the full simulation domain. The moving window is (by default) then G_x x (G_y-1) x G_z in (GPU) size. When stating the simulation, a virtual tracker particle starts with the speed of light and propagates in +y direction. When it reaches the slide point (G_y-1)/G_y * N_y * windowMovePoint (in cells), the last GPU row in y-direction will be removed and placed in front. The simulation continues till the new slide point is reached and the last GPUs are removed and placed in front again. An illustration can be found here

Currently, I do not understand your 5 questions. Could you reformulate them based on the input above?

[sbastrakov]

Okay, so @PrometheusPi and I wrote answers at the same time, hopefully they complement each other. In case you have further questions @cbontoiu , they are welcome.

[cbontoiu]

Thanks for your answers. Very useful. In conclusion I cannot use the moving window with a single GPU, because of G_y > 1. In this case I will try on a cluster when I get the chance.

Meanwhile maybe you can help me with the following error (cudaErrorMisalignedAddress). It happens without the moving window enabled in the .cfg file and only when I reduce the number of mesh cells as for example TBG_gridSize="80 32 40" and

using SuperCellSize = mCT::Int< 8, 8, 8 >;
using GuardSize = typename mCT::shrinkTo< mCT::Int< 1, 1, 1 >, simDim >::type;
static constexpr uint32_t BYTES_EXCHANGE_X = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EXCHANGE_Y = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EXCHANGE_Z = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EDGES = 128 * 1024; // 128 kiB
static constexpr uint32_t BYTES_CORNER = 32 * 1024; // 32 kiB

in the memory.param file.

[sbastrakov]

Yes, sorry for not mentioning it earlier - moving window is inherently a multi-process feature.

Thanks for reporting the error @cbontoiu . This is probably some error on PIConGPU side: either such a configuration should not be allowed, with a clear error message, or there is some alignment bug that was hidden by our standard configurations. We will investigate.

[sbastrakov]

@cbontoiu I created a new issue #3202 for your error report, since this one started as moving window-related. If possible, please provide the details there. In case you feel the moving window questions are currently resolved, please feel free to close this one.

[cbontoiu]

OK, I will try to reproduce the error and send you the archive with all files. Maybe you will need to run it with a single GPU as I do, in order to get that error.

[psychocoderHPC]

Thanks for your answers. Very useful. In conclusion I cannot use the moving window with a single GPU, because of G_y > 1. In this case I will try on a cluster when I get the chance.

@cbontoiu For a while using a moving window with a single GPU is possible by adding the argument -r 2 to PIConGPU. You can add this parameter in the cfg equal to how you setup a plugin parameter. Using -r 2 will allow pushing two MPI ranks to a single GPU and requires to update your mpiexec call or srun in the template tpl to e.g. mpiexec -n 2 <path_to_pic_binary>/picongpu -r 2. Be aware that your visible volume in Y-direction is reduced by one GPU in the openPMD output due to the way how we implemented moving window. On NVIDIA GPU's I observed a slowdown of the time to solution. With AMD GPUs it is running much better.