Unable to run dorado basecaller with CPU.

[VBHarrisN]

Issue Report

Unable to run dorado basecaller with CPU.

Please describe the issue:

Despite attempting to use either the -x or --device flag in the dorado basecaller, dorado keeps throwing c10: error CUDA error: no CUDA-capable device is detected. Please provide a clear and concise description of the issue you are seeing and the result you expect. I cannot get dorado basecaller to use the CPU instead of the GPU. Given that I do not have a CUDA capable card in my computer, and that I am flagging the CPU for use, I do not understand why the error keeps occuring.

Steps to reproduce the issue:

This is the command I am running: dorado basecaller dna_r10.4.1_e8.2_400bps_sup@v5.0.0 ./barcode10 --device "cpu" > test.bam Please list any steps to reproduce the issue. Just attempt to run the basecaller with the the cpu flag.

Run environment:

• Dorado version: 0.7.1
• Dorado command: dorado basecaller
• Operating system: Ubuntu
• Hardware (CPUs, Memory, GPUs): 13th Gen Intel(R) Core(TM) i9, 64GB of RAM, no GPU.
• Source data type (e.g., pod5 or fast5 - please note we always recommend converting to pod5 for optimal basecalling performance): 3 pod5 files which have been successfully basecalled.
• Source data location (on device or networked drive - NFS, etc.): on C: Drive.
• Details about data (flow cell, kit, read lengths, number of reads, total dataset size in MB/GB/TB):
• Dataset to reproduce, if applicable (small subset of data to share as a pod5 to reproduce the issue):

  
  ## Logs

• Please provide output trace of dorado (run dorado with -v, or -vv on a small subset) [2024-06-14 13:49:07.551] [info] > Creating basecall pipeline terminate called after throwing an instance of 'c10::Error' what(): CUDA error: no CUDA-capable device is detected CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Device-side assertions were explicitly omitted for this error check; the error probably arose while initializing the DSA handlers. Exception raised from c10_cuda_check_implementation at /pytorch/pyold/c10/cuda/CUDAException.cpp:44 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f70d4afe9b7 in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::string const&) + 0x64 (0x7f70ce083115 in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const, char const, int, bool) + 0x118 (0x7f70d4ac8958 in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #3: c10::cuda::CUDAKernelLaunchRegistry::CUDAKernelLaunchRegistry() + 0x9d (0x7f70d4ac802d in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #4: c10::cuda::CUDAKernelLaunchRegistry::get_singleton_ref() + 0x44 (0x7f70d4ac81f4 in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #5: c10::cuda::c10_cuda_check_implementation(int, char const, char const, int, bool) + 0x35 (0x7f70d4ac8875 in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #6: c10::cuda::current_device() + 0x2e (0x7f70d4ac89fe in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #7: c10::cuda::getCurrentCUDAStream(signed char) + 0x8d (0x7f70d4ad38ad in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #8: dorado() [0xa3342e] frame #9: dorado() [0xa39bba] frame #10: dorado() [0xa259fd] frame #11: dorado() [0xa30d8a] frame #12: dorado() [0xa0a732] frame #13: dorado() [0xa15461] frame #14: dorado() [0xa147b4] frame #15: dorado() [0x88bf3e] frame #16: dorado() [0x88c7db] frame #17: + 0x1196e380 (0x7f70dba34380 in /home/user/packages/dorado-0.7.1-linux-x64/bin/../lib/libdorado_torch_lib.so) frame #18: + 0x94ac3 (0x7f70c8e3fac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #19: + 0x126850 (0x7f70c8ed1850 in /lib/x86_64-linux-gnu/libc.so.6)

[malton-ont]

Hi @VBHarrisN,

Thanks for reporting this! It appears there is an issue in the transformer model (v5) implementation that is incorrectly attempting to use the CUDA functions even when selecting for CPU. We'll try to get a fix out for this in short order. In the mean time you'll need to use the v4.x models.

I'd also recommend getting a GPU for your system - we don't expect the transformer models to be at all performant in CPU mode.

[iiSeymour]

Fixed in v0.7.2 released just now.