royerlab / ultrack

Cell tracking and segmentation software
https://royerlab.github.io/ultrack
BSD 3-Clause "New" or "Revised" License
64 stars 7 forks source link

Skip CuPy v13.0.* due to broken `ElementwiseKernel` and other core kernels #88

Closed qin-yu closed 4 months ago

qin-yu commented 4 months ago

Without !=13.0.*, timelapse_flow() failed. It is promised to be fixed in v13.1.0 so we avoid all v13.0.*. I tested tracking with flow on the flow field 3D env and believe other gpu env dependencies should also avoid this version.

Related Issues:

qin-yu commented 4 months ago
Computing flow:   0%|                                                                                                                          | 0/9 [01:03<?, ?it/s]
Traceback (most recent call last):
  File "/home/repositories/test-ultrack/track-with-flow.py.py", line 145, in <module>
    main("/home/Datasets/EMBL.tif")
  File "/home/repositories/test-ultrack/track-with-flow.py.py", line 48, in main
    flow = timelapse_flow(
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/ultrack/imgproc/flow.py", line 632, in timelapse_flow
    flow_field(
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/ultrack/imgproc/flow.py", line 205, in flow_field
    scaled_source = _interpolate(
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/ultrack/imgproc/flow.py", line 71, in _interpolate
    blurred = ndi.gaussian_filter(
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupyx/scipy/ndimage/_filters.py", line 373, in gaussian_filter
    return _run_1d_correlates(input, list(zip(sigmas, orders)), get, output,
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupyx/scipy/ndimage/_filters.py", line 206, in _run_1d_correlates
    return _filters_core._run_1d_filters(
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupyx/scipy/ndimage/_filters_core.py", line 105, in _run_1d_filters
    fltr(input, arg, axis, output, mode, cval, origin)
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupyx/scipy/ndimage/_filters.py", line 114, in correlate1d
    return _correlate_or_convolve(input, weights, output, mode, cval, origins)
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupyx/scipy/ndimage/_filters.py", line 177, in _correlate_or_convolve
    output = _filters_core._call_kernel(kernel, input, weights, output,
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupyx/scipy/ndimage/_filters_core.py", line 150, in _call_kernel
    kernel(*args)
  File "cupy/_core/_kernel.pyx", line 920, in cupy._core._kernel.ElementwiseKernel.__call__
  File "cupy/_core/_kernel.pyx", line 945, in cupy._core._kernel.ElementwiseKernel._get_elementwise_kernel
  File "cupy/_util.pyx", line 64, in cupy._util.memoize.decorator.ret
  File "cupy/_core/_kernel.pyx", line 728, in cupy._core._kernel._get_elementwise_kernel
  File "cupy/_core/_kernel.pyx", line 82, in cupy._core._kernel._get_simple_elementwise_kernel_from_code
  File "cupy/_core/core.pyx", line 2254, in cupy._core.core.compile_with_cache
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 484, in _compile_module_with_cache
    return _compile_with_cache_cuda(
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 562, in _compile_with_cache_cuda
    ptx, mapping = compile_using_nvrtc(
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 319, in compile_using_nvrtc
    return _compile(source, options, cu_path,
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 284, in _compile
    options, headers, include_names = _jitify_prep(
  File "/home/miniconda3/envs/ultrack-plant-seg-fixcupy/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 233, in _jitify_prep
    jitify._init_module()
  File "cupy/cuda/jitify.pyx", line 212, in cupy.cuda.jitify._init_module
  File "cupy/cuda/jitify.pyx", line 233, in cupy.cuda.jitify._init_module
  File "cupy/cuda/jitify.pyx", line 209, in cupy.cuda.jitify._init_cupy_headers
  File "cupy/cuda/jitify.pyx", line 192, in cupy.cuda.jitify._init_cupy_headers_from_scratch
  File "cupy/cuda/jitify.pyx", line 264, in cupy.cuda.jitify.jitify
RuntimeError: Runtime compilation failed
JoOkuma commented 4 months ago

Thank you @qin-yu!