Struct-of-arrays based short-range loop

[RudolfWeeber]

Goal

significantly faster calculation of short range force/pressure/energy by using a more cache-friendly data structure

Background

• ESPResSo currently stores particles as a struct Particle containing all the attributes (pos, velocity, charge, mass) of a single particle.
• Many particles are then stored as arrays (vectors) of Particle structs (array-of-struct).
• This layout makes individual particles easy to handle, as the Particle struct can be passed to a function which does something with a single particle.
• However, short range kernels, which need only a small subset of properties, but for several particles, are slow. This is, because memory reads into the cache happen in blocks. With the array-of-struct layout, the position of two consecutive particles is further apart than the size of the block. So, a lot of the possible memory bandwidth gets wasted.
• It would be much faster, if the position of all particles would be stored next to each other in memory, such that a read of a single block would result in may particle positions being in the cache.
• This layout is called struct-of-arrays.

Cabana and Kokkos

• Cabana is a library that provides infrastructure for particle codes, e.g. neighbor (Verlet) lists and mechanisms to run kernels on them in parallel. It makes use of Kokkos for performance portability
• Kokkos is a performance portability layer, which let's one write code that runs both, on the CPU and the GPU.

Steps

• [X] Prototype of struct-of-arrays-based force calculation for linked-cells
  • This was done for LJ. It is maybe 2x faster for an LJ cutoff of 2.5 sigma.
• [X] Separate short-range forces by category
  • categories:
    • central, radial (e.g. Lennard-Jones, Hertzian, soft sphere)
    • central radial with charge (e.g., P3M/Ewald real-space force, Coulomb force)
    • dipolar
    • others
  • These differ in which particle properties are required so depending on what is active, only thoe properties have to be provided as arrays.
• [ ] Add Kokkos and Cabana as optional dependency to ESPResSo in CMake.
• [ ] Add Kokkos and Cabana to CI containers
• [ ] Implement copying particle data from ESPResSo to Cabana and resulting forces back to ESPResSo
• [ ] Fulfil the Lennard-Jones tet for a single particle species by using the copied particle data, Cabana's neighbor list example with a user specified cut-off, and ESPResSo's Lennard-Jones kernel
• [ ] Use ESPResSo's maximal short range cutoff instead of a user provided one
• [ ] Adapt this to handle multiple particle types and other central forces including Coulom-P3M short range kernel
• [ ] Use ESPResSo's customized criteroin for decising whether pairs should be added to the neighbor list
• [ ] Investigate GPU off-loading of short range forces

[jngrad]

For a graphical representation of the proposed changes to the data structure, see Fig. 2 and section "3.1. Improved data layout" of Vance et al. 2023 (doi:10.1016/j.cpc.2023.108760).

[junghans]

/CC @streeve, to make you aware that @espressomd looking at Cabana.