ROCM HIP / Vulkan support for GPU acceleration

[DistantThunder]

Is your feature request related to a problem? Please describe: Right now, it is not possible to use this library on AMD hardware because it is ABI-linked to either the CPU or CUDA version of Torch. It doesn't appear to be possible to build it with just replacing the torch version with rocm-enabled torch unlike what usually works for other ML projects.

Describe the solution you would like: Be able to have rocm enabled build of this library available or at least implement HIP/Vulkan support so that it can be built from source.

Describe the alternatives you have considered: I have none at this moment besides running on the CPU.

Additional Context: ROCM 5.6.x / Radeon 7900 XTX working with most other tools relying on Torch.

[dc-dc-dc]

From what I can tell, the only cuda specific code is under fairseq2n/src/fairseq2n/generation/ngram_repeat_block.cu, if this is the case I can take a look at creating a hip specific version and adding ROCM targets in the appropriate cmake files

[cbalioglu]

@DistantThunder, @dc-dc-dc sorry for my very late reply. The past couple of months have been very busy; just catching up with many backlog items. #270 is a similar ask about supporting backends beyond CUDA. I am supportive of having additional backends (in particular HIP and MLX), the only tricky part is making sure that they do not lag behind CUDA. We will land quite a few CUDA kernels soon and TBH I don't have any experience with HIP (neither most other folks in FAIR), so we should figure out a way to properly test them and ensure that they have parity with CUDA (both accuracy and performace-wise). I will ask internally whether we can get access to some AMD hardware. In the meantime, like I mentioned in @270, if you have a PR (sorry @dc-dc-dc just saw that you closed your pending PR) that would add preliminary support for HIP, please feel free to send it (and add me as reviewer)

[dc-dc-dc]

If someone wants to continue forward from where I left the HIP PR feel free.

I have started contributing to the mlx repo, so can contribute with adding support for fairseq. Would be awesome seeing fairseq2 have native support for mlx and unlocking access to some of the sota models coming from fair.