search

Alpine-DAV / ascent

A flyweight in situ visualization and analysis runtime for multi-physics HPC simulations
https://alpine-dav.github.io/ascent/
Other
195 stars 66 forks source link

Is `-munsafe-fp-atomics` safe on AMD GPUs? #1248

Open pgrete opened 9 months ago

pgrete commented 9 months ago

How many device atomics are used in Ascent and it's software stack and is the internal programming/memory model safe with regard to this, see https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#floating-point-fp-atomic-operations-and-coarse-fine-grained-memory-allocations ?

We've been using Kokkos::ScatterViews on AMD and without hardware atomics performance is >100x worse than with so I'm wondering if it's safe to globally turn on when compiling Ascent (and not just for our downstream code that links against Ascent).

cyrush commented 9 months ago

@pgrete We are using atomics and I believe our allocations are coarse grained - which would be compatible with -munsafe-fp-atomics.

I looked at the allocs we use with RAJA + I asked the VTK-m team to check. Since VTK-m is are using Kokkos and default device allocations are coarse grained, I would expect them to be fine.

We haven't done any explicit performance testing with this flag (unless VTK-m or Kokkos is turning it on for us)

pgrete commented 9 months ago

Great, thanks, I'll give this a try. Feel free to close the issue once confirmed with the other libs.