Convert cuda/nvshmem api to hip/rocshmem

[dalg24-jenkins]

Can one of the admins verify this patch?

[Rombur]

OK to test

[janciesko]

Please run apply-clang-format to pass CI.

[BKP]

Please run apply-clang-format to pass CI.

Done.

I needed to change the version in the apply-clang-format file from version 8 to version 15 (on Frontier).

There were files in the other directories (shmem, nvshmem) that the apply-clang-format script updated. I only included the files in the rocshmem subdirectory in the pushed changesets.

[janciesko]

This still has a small formatting missmatch with our clang-format script.

[janciesko]

Let's make this merge contingent on passing unit tests on Crusher

[BKP]

Tests pass on both Crusher and Frontier.

[janciesko]

Thank you.