NVRTC compilation error

[lacker]

When attempting to run find_et_serial I get an NVRTC compilation error:

Traceback (most recent call last):
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 554, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 108, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 120, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 58, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lacker/astro/./hyper.py", line 7, in <module>
    hits = hyperseti.find_et_serial(
  File "/home/lacker/astro/hyperseti/hyperseti.py", line 533, in find_et_serial
    dedopp, metadata, hits = run_pipeline(d, h5.metadata, *args, **kwargs)
  File "/home/lacker/astro/hyperseti/hyperseti.py", line 379, in run_pipeline
    _peaks = hitsearch(dedopp, metadata, threshold=_threshold, min_fdistance=min_fdistance, min_ddistance=min_ddistance)
  File "/home/lacker/astro/hyperseti/hyperseti.py", line 274, in hitsearch
    intensity, fcoords, dcoords = prominent_peaks(dedopp_gpu, min_xdistance=min_fdistance, min_ydistance=min_ddistance, threshold=threshold)
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupyimg/skimage/feature/peak.py", line 385, in _prominent_peaks
    img_max = ndi.maximum_filter1d(
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupyimg/scipy/ndimage/filters.py", line 1475, in maximum_filter1d
    return _min_or_max_1d(input, size, axis, output, mode, cval, origin, "max")
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupyimg/scipy/ndimage/filters.py", line 1505, in _min_or_max_1d
    return _filters_core._call_kernel(
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupyimg/scipy/ndimage/_filters_core.py", line 152, in _call_kernel
    kernel(*args)
  File "cupy/core/_kernel.pyx", line 821, in cupy.core._kernel.ElementwiseKernel.__call__
  File "cupy/core/_kernel.pyx", line 846, in cupy.core._kernel.ElementwiseKernel._get_elementwise_kernel
  File "cupy/_util.pyx", line 59, in cupy._util.memoize.decorator.ret
  File "cupy/core/_kernel.pyx", line 639, in cupy.core._kernel._get_elementwise_kernel
  File "cupy/core/_kernel.pyx", line 37, in cupy.core._kernel._get_simple_elementwise_kernel
  File "cupy/core/_kernel.pyx", line 60, in cupy.core._kernel._get_simple_elementwise_kernel
  File "cupy/core/core.pyx", line 1882, in cupy.core.core.compile_with_cache
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 373, in compile_with_cache
    return _compile_with_cache_cuda(
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 440, in _compile_with_cache_cuda
    ptx, mapping = compile_using_nvrtc(
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 211, in compile_using_nvrtc
    return _compile(source, options, cu_path,
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 195, in _compile
    ptx, mapping = prog.compile(options, log_stream)
  File "/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 565, in compile
    raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: /home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/core/include/cupy/complex/complex.h(94): warning: __host__ annotation is ignored on a function("complex") that is explicitly defaulted on its first declaration

/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/core/include/cupy/complex/complex.h(94): warning: __device__ annotation is ignored on a function("complex") that is explicitly defaulted on its first declaration

/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/core/include/cupy/complex/complex.h(101): warning: __host__ annotation is ignored on a function("complex") that is explicitly defaulted on its first declaration

/home/lacker/miniconda3/envs/astro/lib/python3.9/site-packages/cupy/core/include/cupy/complex/complex.h(101): warning: __device__ annotation is ignored on a function("complex") that is explicitly defaulted on its first declaration

/tmp/tmpyifgeq6s/8bc753d0a082f7bbfe75e4bc152b5195_2.cubin.cu(10): catastrophic error: cannot open source file "type_traits"

1 catastrophic error detected in the compilation of "/tmp/tmpyifgeq6s/8bc753d0a082f7bbfe75e4bc152b5195_2.cubin.cu".
Compilation terminated.

Maybe there is a dependency I am missing, or a particular cupy version?

[lacker]

Okay, I believe the error here is due to https://github.com/cupy/cupy/issues/4875 . This is a problem in at least version 8.6 of cupy which is currently the most recent release. I managed to get the tests running when I used cudatoolkit 11.0 and the cupy beta release wheel version cupy-cuda110==9.0.0b3. So for anyone who runs into this in the future, I recommend those as a working combination.

[telegraphic]

Glad you figured this out! (That one-line change looks pretty innocuous, but with disastrous effect)