Simulation slows down when particle boundary buffers are active

[roelof-groenewald]

When the scraped particle boundary buffers are activated there appears to be a continual decrease in simulation performance, even when the buffers are periodically cleared. As a example of this: I am running a 3d simulation on 4 A100 GPUs with the embedded boundary scraped particle buffer on (since I am reflecting particles from that boundary). The buffer is cleared after every step. At the start of the simulation it progresses at ~0.57 steps/sec but by step 20000 the performance have degraded to 0.42 steps/sec even though the number of particles in the simulation remain fixed. Looking at the tiny profile output for runs up to different timesteps, it looks like the time spent in ParticleBoundaryBuffer::gatherParticles::filterTransformEB increases disproportionately to the number of steps taken.

I believe this issue has been previously noted by @dpgrote as well (but I couldn't quickly find a link to that discussion).

cc @RemiLehe this would be important to clear up if the particle boundary buffers will be used for SEE and other PMI models. cc @atmyers

[roelof-groenewald]

One thing I can check is to compare the number of particles in the buffer to the time spent in gathering particles just to confirm that the simulation slow down is not due to more particles ending up on the boundary at later times.

[dpgrote]

In the case where I was seeing a slow down the lost particle buffer was not being cleared so the number of particles in the buffer continuously increased.