CUDA_ERROR_OUT_OF_MEMORY when making a projected image of a snapshot

[MatthieuSchaller]

I am running this example: https://swiftsimio.readthedocs.io/en/latest/visualisation/projection.html#example

on a snapshot with 256^3 gas particles at z=1. Box is 150Mpc.

I get the following error with python 3.9.1 on a cosma-8 login node:

Traceback (most recent call last):
  File "/cosma8/data/dp004/jlvc76/SWIFT/master/swiftsim/examples/MHD_box/snapshots/plot.py", line 1, in <module>
    from swiftsimio import load
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/swiftsimio/__init__.py", line 11, in <module>
    import swiftsimio.visualisation as visualisation
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/swiftsimio/visualisation/__init__.py", line 5, in <module>
    from .projection import scatter, project_gas, project_gas_pixel_grid
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/swiftsimio/visualisation/projection.py", line 28, in <module>
    from swiftsimio.visualisation.projection_backends import backends, backends_parallel
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/swiftsimio/visualisation/projection_backends/__init__.py", line 45, in <module>
    from swiftsimio.visualisation.projection_backends.gpu import scatter as gpu
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/swiftsimio/visualisation/projection_backends/gpu.py", line 68, in <module>
    def scatter_gpu(
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/decorators.py", line 115, in _jit
    disp.compile(argtypes)
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 794, in compile
    kernel = _Kernel(self.py_func, argtypes, **self.targetoptions)
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 94, in __init__
    self.cooperative = 'cudaCGGetIntrinsicHandle' in lib.get_asm_str()
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/codegen.py", line 110, in get_asm_str
    return self._join_ptxes(self._get_ptxes(cc=cc))
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/codegen.py", line 114, in _get_ptxes
    ctx = devices.get_context()
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 220, in get_context
    return _runtime.get_or_create_context(devnum)
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 138, in get_or_create_context
    return self._get_or_create_context_uncached(devnum)
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 155, in _get_or_create_context_uncached
    return self._activate_context_for(0)
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 177, in _activate_context_for
    newctx = gpu.get_primary_context()
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 664, in get_primary_context
    driver.cuDevicePrimaryCtxRetain(byref(hctx), self.id)
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 320, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/cosma/home/jlvc76/.local/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 388, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuDevicePrimaryCtxRetain results in CUDA_ERROR_OUT_OF_MEMORY

[JBorrow]

Why are you using cuda?

[JBorrow]

Are you using backend="gpu" or no?

[MatthieuSchaller]

No cuda. Just the base example on the webpage.

[JBorrow]

Can you show me what modules you're using? It seems it is getting a CUDA OOM error on just checking for the device. What happens if you just do the imports instead of actually trying to do anything?

[MatthieuSchaller]

Only module loaded is python/3.9.1-C8

[JBorrow]

I can't reproduce this on my end. I think it's likely a problem with your numba installation, rather than something swiftsimio specific. Try uninstalling and reinstalling numba.

[MatthieuSchaller]

As usual with me and python, it must be a keyboard-chair interface issue.