search

AMReX-Codes / amrex

AMReX: Software Framework for Block Structured AMR
https://amrex-codes.github.io/amrex
Other
539 stars 344 forks source link

Plans for AMD GPU support #345

Closed crtrott closed 5 years ago

crtrott commented 5 years ago

Hi,

I was wondering what your current plans are to support AMD GPUs. Is this something you thought about how to do?

asalmgren commented 5 years ago

We're currently exploring three approaches to supporting applications on GPUs -- CUDA, OpenACC and OpenMP.

Do you have any reasons to believe the latter two won't be available on AMD GPUs?

On Thu, Oct 18, 2018 at 2:39 PM, Christian Trott notifications@github.com wrote:

Hi,

I was wondering what your current plans are to support AMD GPUs. Is this something you thought about how to do?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/amrex/issues/345, or mute the thread https://github.com/notifications/unsubscribe-auth/AKJPYi1oefR0UoIf22UYOXS9At4GryiBks5umPT-gaJpZM4XvSTE .

crtrott commented 5 years ago

OpenACC as a production solution is something I wouldn't bet on. The only vendor directly supporting this is NVIDIA after all through PGI, and while there is occasional talk about this being supported for AMD I don't think it is on the roadmaps. I am not sure what the production vendor compiler story on OpenMP is. So far the mainline compilers doing OpenMP for GPUs in a setting where you can have a contract to fix bugs are IBM and Cray. There is Clang integration going on, but since this is a community compiler I am not sure you can get anyone to contractual committing to fix bugs as necessary as we have been doing with IBM or Intel for the big machines.

But maybe I am wrong about what AMD has put on their actual roadmaps.

asalmgren commented 5 years ago

That's good to know.

What is the kokkos strategy for AMD GPUs?

On Thu, Oct 18, 2018 at 3:27 PM, Christian Trott notifications@github.com wrote:

OpenACC as a production solution is something I wouldn't bet on. The only vendor directly supporting this is NVIDIA after all through PGI, and while there is occasional talk about this being supported for AMD I don't think it is on the roadmaps. I am not sure what the production vendor compiler story on OpenMP is. So far the mainline compilers doing OpenMP for GPUs in a setting where you can have a contract to fix bugs are IBM and Cray. There is Clang integration going on, but since this is a community compiler I am not sure you can get anyone to contractual committing to fix bugs as necessary as we have been doing with IBM or Intel for the big machines.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/amrex/issues/345#issuecomment-431185189, or mute the thread https://github.com/notifications/unsubscribe-auth/AKJPYm6AEh-3RmsrizbA991uD5laHmEOks5umQBEgaJpZM4XvSTE .

crtrott commented 5 years ago

Originally AMD recommended using the HCC native frontend directly. This is C++ AMP based interface wise. We have a backend for this, which passes practically the whole Kokkos UnitTest suite, but our applications expose a number of compiler bugs which are show stoppers (we explicitly worked around those compiler bugs in Kokkos internals). Recently AMD recommended a shift to using the HIP front end which apparently is a bit more robust at this point. We are starting a new backend based on that (AMD is putting staff on that work too) but keep the other one around in order to help with compiler maturation. The reason is that strategically the native HCC frontend is supposed to getting better aligned with the evolving C++ standard.

crtrott commented 5 years ago

Btw. this question came up in discussion with some folks here at Sandia. They were trying to use Trillions for a geometric multi grid code which also uses BoxLib or going forward AMReX. From our point of view Trillions is probably not the right thing for that, since it is intended for unstructured problems - but as part of that discussion we were talking about how we are planning to support the possible ExaScale machine contenders (Intel,NVIDIA,AMD,ARM,IBM).

ax3l commented 5 years ago

Are you considering HIP or SYCL? At least those are candidates we are looking into with Alpaka. At least for LLVM/Clang, the signs are good for both on a few-year timescale.

Will be interesting what will be centrally available for Frontier.

WeiqunZhang commented 5 years ago

We are considering HIP. SYCL is not listed on Frontier Spec Sheet. https://www.olcf.ornl.gov/wp-content/uploads/2019/05/frontier_specsheet_v4.pdf