Enable global particle redistribution based on the grid

streeve commented 7 months ago

While it should often be avoided, global communication is occasionally needed in applications. This uses the background grid to determine the rank to send any particle to (including removing those outside the global domain).

Still to do:

Consider moving the migrate functionality to a free function or separate class
Consider naming scheme - this primarily determines comm destinations, to then create the communication plan and distributor and eventually migrate
In this PR or separately, add an interface to extract only those particles that don't "belong" on the current rank from the migration destinations and ghost them instead

streeve commented 7 months ago

@kwitaechong can you test this as a replacement for your custom migrateAllToAll?

streeve commented 7 months ago

Currently relying on KOKKOS_CLASS_LAMBDA without requiring C++20 causing the CI failures

streeve commented 6 months ago

retest this please.

streeve commented 6 months ago

@kwitaechong can you test on Summit or Frontier to find where the device failure is occurring?

streeve commented 6 months ago

Manually checked that tests pass with SYCL. Should be ready to merge once it's removed from Cuda-SYCL tests

streeve commented 5 months ago

I/O timeouts unrelated

ECP-copa / Cabana

Enable global particle redistribution based on the grid #707