search

DUNE / larnd-sim

Simulation framework for a pixelated Liquid Argon TPC
Apache License 2.0
10 stars 26 forks source link

--n_events option in simulate_pixels.py not working #193

Open krwood opened 6 months ago

krwood commented 6 months ago

Specifying certain values for the --n_events option in simulate_pixels.py crashes the simulation. When running on a file with 200 events (= NuMI spills in 2x2), it runs fine when not specifying --n_events; however, when specifying --n_events = 1, there is a cuda illegal address error:

Loading track segments...Selecting only the first 1 events for simulation.
Data preparation time: 2.73 s
******************
RUNNING SIMULATION
******************
Quenching electrons... 0.42 s
Drifting electrons... 0.20 s

Calculating optical responses... 34.23 s
Simulating batches...:   0%|                              | 0/1 [00:00<?, ?it/s]ERROR:numba.cuda.cudadrv.driver:Call to cuMemcpyDtoH results in UNKNOWN_CUDA_ERROR
Simulating batches...:   0%|                              | 0/1 [00:05<?, ?it/s]
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/commit/clean/larnd-sim/cli/simulate_pixels.py", line 1143, in <module>
    fire.Fire(run_simulation)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/commit/clean/larnd-sim/cli/simulate_pixels.py", line 993, in run_simulation
    light_sim.sum_light_signals[BPG, TPB](
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 539, in __call__
    return self.dispatcher.call(args, self.griddim, self.blockdim,
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 675, in call
    kernel.launch(args, griddim, blockdim, stream, sharedmem)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/dispatcher.py", line 371, in launch
    wb()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/args.py", line 62, in <lambda>
    retr.append(lambda: devary.copy_to_host(self.value, stream=stream))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/devices.py", line 232, in _require_cuda_context
    return fn(*args, **kws)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/devicearray.py", line 277, in copy_to_host
    _driver.device_to_host(hostary, self, self.alloc_size,
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 3145, in device_to_host
    fn(host_pointer(dst), device_pointer(src), size, *varargs)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemcpyDtoH results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1698, in core
    dealloc.add_item(module_unload, handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1696, in module_unload
    driver.cuModuleUnload(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1698, in core
    dealloc.add_item(module_unload, handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1696, in module_unload
    driver.cuModuleUnload(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1698, in core
    dealloc.add_item(module_unload, handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1696, in module_unload
    driver.cuModuleUnload(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuModuleUnload results in UNKNOWN_CUDA_ERROR
ERROR:numba.cuda.cudadrv.driver:Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 2173, in deref
    mem.free()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1962, in free
    self._finalizer()
  File "/global/common/software/nersc/pm-2022q3/sw/python/3.9-anaconda-2021.11/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1614, in core
    deallocations.add_item(driver.cuMemFree, ptr, size)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1180, in add_item
    self.clear()
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 1191, in clear
    dtor(handle)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuMemFree results in UNKNOWN_CUDA_ERROR
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:
Traceback (most recent call last):
  File "/global/cfs/cdirs/dune/users/kwood/ndlar_flow_MR5/larndsim.venv/larndsim.venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Original exception was:
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 234, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:
Error in sys.excepthook:

Original exception was:

This has been observed on both Perlmutter and SLAC machines.

liviocali commented 6 months ago

n_events=10 works

sam-fogarty commented 5 months ago

@krwood @liviocali Is this still an issue? I got nevents=1 and nevents=5 to run on 2x2_mod2mod_variation config and a miniRun4 file. (I haven't tested on the 2x2 config yet as I was running into an unrelated error.)