SyneRBI / SIRF

Main repository for the CCP SynerBI software
http://www.ccpsynerbi.ac.uk
Other
58 stars 29 forks source link

Need to disable CUDA test on GHA (also for the SuperBuild) #1278

Closed KrisThielemans closed 1 month ago

KrisThielemans commented 1 month ago

GitHub Actions doesn't have the CUDA run-time, so testing the CUDA RDP fails with sirf.Utilities.error: ??? "'\\nERROR: CUDA failed to allocate memory in compute_gradient kernel execution: CUDA driver version is insufficient for CUDA runtime version\\n' This occurs in the build (gpu) docker job of the SB, see e.g. https://github.com/SyneRBI/SIRF-SuperBuild/actions/runs/9865000620/job/27241074266#step:9:29413

Actual relevant lines are https://github.com/SyneRBI/SIRF/blob/cb8a9981b142006c3b01cf9264f51882afdad12a/src/xSTIR/pSTIR/tests/tests_qp_lc_rdp.py#L64-L65

In STIR, I've disabled the tests by checking in CMake if nvida-smiis found https://github.com/UCL/STIR/blob/09638a9f01ff674118f1d584dbeba604e2435bf9/CMakeLists.txt#L209-L213 and then passing an option to the relevant test https://github.com/UCL/STIR/blob/09638a9f01ff674118f1d584dbeba604e2435bf9/src/recon_test/CMakeLists.txt#L152-L154

For SIRF, we're using pytest. @casperdcl any suggestions

paskino commented 1 month ago

@casperdcl set up a self hosted runner for CIL's CI where GPU is required. Similar thing could be done here, I believe.

Short of that, we run nvidia-smi during pytest to see if the machine where the tests run has a GPU that works:

try:
    subprocess.check_output('nvidia-smi')
    has_nvidia = True
except:
    if os.environ.get("TESTS_FORCE_GPU", ""):
        raise ImportError
    has_nvidia = False