Support for Multiple ROCM versions

[d3v-null]

• Currently this repo's bindings were generated from rocm version 5.4.22804, and don't play nicely with rocm > 6.
• Because of how immature the ecosystem is, various versions of ROCM are incompatible with different hardware. e.g. Setonix appears to play nicely on rocm > 6, but segfaults on rocm < 6

As @cjordan has pointed out, there are two approaches:

• generating bindings for specific versions of hip/rocm and making them available as different cargo features
• introduce a bindgen feature that has the bindings generated at build time, but this relies on libclang being available and may need to some massaging done to the bindings immediately after they're generated, e.g. https://github.com/coreylowman/cudarc/

[cjordan]

Thanks Dev. My time unfortunately strained, but working on Rust sparks joy :) I'll try to make progress on this on the weekend or coming weekends.

[d3v-null]

All good, looks like build.rs already had a bindgen feature, so that basically covers my needs. All I need to do is figure out some logic to provide all of the magic incantations to clang for each version of rocm, because placement of files changes a lot between versions:

export targets="hip_runtime_api.h stddef.h libclang.so"
for ROCM_VER in 5.4.6 5.6.1 5.7.3 6.0.2 6.1; do
    echo $ROCM_VER;
    docker run -w /opt/rocm quay.io/pawsey/rocm-mpich-base:rocm${ROCM_VER}-mpich3.4.3-ubuntu22 \
        /bin/bash -c "for q in $targets; do find -L . -name \$q; done"

done

5.4.6
./include/hip/hip_runtime_api.h
./hip/include/hip/hip_runtime_api.h
./llvm/lib/clang/15.0.0/include/stddef.h
./include/hipify/stddef.h
./llvm/lib/libclang.so
5.6.1
./include/hip/hip_runtime_api.h
./hip/include/hip/hip_runtime_api.h
./llvm/lib/clang/16.0.0/include/stddef.h
./include/hipify/stddef.h
./llvm/lib/libclang.so
5.7.3
./include/hip/hip_runtime_api.h
./hip/include/hip/hip_runtime_api.h
./llvm/lib/clang/17.0.0/include/stddef.h
./include/hipify/stddef.h
./llvm/lib/libclang.so
6.0.2
./include/hip/hip_runtime_api.h
./lib/llvm/lib/clang/17.0.0/include/stddef.h
./llvm/lib/clang/17.0.0/include/stddef.h
./include/hipify/stddef.h
./lib/llvm/lib/libclang.so
./llvm/lib/libclang.so
6.1
./include/hip/hip_runtime_api.h
./lib/llvm/lib/clang/17/include/stddef.h
./llvm/lib/clang/17/include/stddef.h
./include/hipify/stddef.h
./lib/llvm/lib/libclang.so
./llvm/lib/libclang.so

When I'm done, the tests should pass on all rocm versions:

for ROCM_VER in 5.4.6 5.6.1 5.7.3 6.0.2 6.1; do
    echo $ROCM_VER;
    docker run -it \
        -v "$PWD:/hip-sys" \
        -w /hip-sys \
        quay.io/pawsey/rocm-mpich-base:rocm${ROCM_VER}-mpich3.4.3-ubuntu22 \
        /bin/bash -c "\
        export RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:\$PATH && \
        mkdir -m755 \$RUSTUP_HOME \$CARGO_HOME && \
        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile=minimal --component=rustfmt --default-toolchain=1.74 && \
        . /opt/cargo/env && \
        cargo install 'bindgen-cli@0.69.4' --quiet && \
        hip-runtime-sys/bindgen.sh && \
        cargo test --features=bindgen"

done

[d3v-null]

I think I've nearly got it. https://github.com/d3v-null/hip-sys/tree/rocm6

It passes the test in that it succesfully compiles in all of those containers, but because of some other issue, the test iteslf fails to initialize rocm rt:

thread 'test_hip_init' panicked at tests/integration_tests.rs:6:5:
assertion `left == right` failed
  left: hipErrorInvalidDevice
 right: hipSuccess
stack backtrace:
   0: rust_begin_unwind
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:72:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:270:5
   4: integration_tests::test_hip_init
             at ./tests/integration_tests.rs:6:5
   5: integration_tests::test_hip_init::{{closure}}
             at ./tests/integration_tests.rs:4:20
   6: core::ops::function::FnOnce::call_once
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

failures:
    test_hip_init

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s

this passes on my local machine with rocm 6.

I added --device=/dev/dri to docker run, and rocm-smi can see my device, but it just won't give a good explanation why it can't initialize rocm rt.

What do you reckon?

[cjordan]

Have you found this page? Maybe it helps

https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html