Megalodon is a research command line tool to extract high accuracy modified base and sequence variant calls from raw nanopore reads by anchoring the information rich basecalling neural network output to a reference genome/transriptome.
Other
196
stars
30
forks
source link
CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:232: CUDA_ERROR_INVALID_DEVICE #351
hello,
I followed the instruction and run the megalodon (2.4.1) and guppy_basecall_server (6.0.1+652ffd1) with the GPU support. But it always reported error information. Although there were some similar issues that others have encountered, I can not solve my problem through their discussion. So I have no choice but to ask for additional help. Attached below is my demo in scripts:
[10:17:00] Running Megalodon version 2.4.1
[10:17:00] Loading guppy basecalling backend
ERROR: Guppy server initialization failed. See guppy logs in [--output-directory] for more details.
Try running the guppy server initialization command found in log.txt in order to pinpoint the source of this issue.
ERROR: Guppy server initialization failed. See guppy logs in [--output-directory] for more details.
Try running the guppy server initialization command found in log.txt in order to pinpoint the source of this issue.
this is the guppy_server log file:
2023-08-08 10:17:01.473438 [guppy/message] ONT Guppy basecall server software version 6.0.1+652ffd1, client-server API version 10.0.0
log path: /public/home/zenglingsen/01.data/01.ONT_data/02.ONT_test/AW/FAST5_PASS/out/guppy_log
chunk size: 2000
chunks per runner: 512
max queued reads: 2000
num basecallers: 4
num socket threads: 2
max returned events: 50000
gpu device: cuda:1
kernel path:
runners per device: 4
2023-08-08 10:17:01.475945 [guppy/info] crashpad_handler not supported on this platform.
2023-08-08 10:17:01.478127 [guppy/info] Listening on port ipc:///tmp/3763-2667-29cb-f0a3.
2023-08-08 10:17:01.607464 [guppy/error] CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:232: CUDA_ERROR_INVALID_DEVICE. Error initialising basecall server using port: ipc://auto. Aborting.
2023-08-08 10:17:01.608638 [guppy/message] The basecall server has shut down successfully.
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
I have followed the reasonable steps through discussion like Guppy can't run on GPU with trained model #46, error while running guppy 4.4.1 on GPU mode #6, and so on. But it does not work.
Any advice and comment are greatful. Thank you very much.
hello, I followed the instruction and run the megalodon (2.4.1) and guppy_basecall_server (6.0.1+652ffd1) with the GPU support. But it always reported error information. Although there were some similar issues that others have encountered, I can not solve my problem through their discussion. So I have no choice but to ask for additional help. Attached below is my demo in scripts:
megalodon \ ~/01.data/01.ONT_data/02.ONT_test/AW/FAST5_PASS \ --guppy-params "-d /public/home/zenglingsen/04.software/03.Guppy/rerio/basecall_models/" \ --guppy-server-path /public/home/zenglingsen/04.software/03.Guppy/ont-guppy/bin/guppy_basecall_server \ --guppy-config res_dna_r941_prom_modbases_5mC_CpG_v001.cfg \ --outputs basecalls mappings mod_mappings mods per_read_mods \ --reference /public/home/zenglingsen/01.data/03.Reference/GCF_000003025.6_Sscrofa11.1_genomic.fna \ --mod-motif m CG 0 \ --output-directory /public/home/zenglingsen/01.data/01.ONT_data/02.ONT_test/AW/FAST5_PASS/out \ --overwrite \ --devices 0 \ --processes 8 \the error file showed:
[10:17:00] Running Megalodon version 2.4.1 [10:17:00] Loading guppy basecalling backend
this is the content in the log.txt:
[10:17:00] Running Megalodon version 2.4.1 DBG 10:17:00 : Command: """/public/home/zenglingsen/04.software/02.Anaconda/Or/envs/pytorch/bin/megalodon /public/home/zenglingsen/01.data/01.ONT_data/02.ONT_test/AW/FAST5_PASS --guppy-params -d /public/home/zenglingsen/04.software/03.Guppy/rerio/basecall_models/ --guppy-server-path /public/home/zenglingsen/04.software/03.Guppy/ont-guppy/bin/guppy_basecall_server --guppy-config res_dna_r941_prom_modbases_5mC_CpG_v001.cfg --outputs basecalls mappings mod_mappings mods per_read_mods --reference /public/home/zenglingsen/01.data/03.Reference/GCF_000003025.6_Sscrofa11.1_genomic.fna --mod-motif m CG 0 --output-directory /public/home/zenglingsen/01.data/01.ONT_data/02.ONT_test/AW/FAST5_PASS/out --overwrite --devices 1 --processes 8""" --- MainProcess-MainThread megalodon.py:1793 [10:17:00] Loading guppy basecalling backend DBG 10:17:00 : Guppy version: "6.0.1" --- MainProcess-MainThread backends.py:939 DBG 10:17:00 : Pyguppy version: "6.0.1" --- MainProcess-MainThread backends.py:940 DBG 10:17:00 : guppy server init command: "/public/home/zenglingsen/04.software/03.Guppy/ont-guppy/bin/guppy_basecall_server -p auto -l /public/home/zenglingsen/01.data/01.ONT_data/02.ONT_test/AW/FAST5_PASS/out/guppy_log -c res_dna_r941_prom_modbases_5mC_CpG_v001.cfg --quiet --post_out -x cuda:1 -d /public/home/zenglingsen/04.software/03.Guppy/rerio/basecall_models/" --- MainProcess-MainThread backends.py:1018 DBG 10:17:01 : Found guppy log file: /public/home/zenglingsen/01.data/01.ONT_data/02.ONT_test/AW/FAST5_PASS/out/guppy_log/guppy_basecall_server_log-2023-08-08_10-17-01.log --- MainProcess-MainThread backends.py:1033
this is the guppy_server log file:
2023-08-08 10:17:01.473438 [guppy/message] ONT Guppy basecall server software version 6.0.1+652ffd1, client-server API version 10.0.0 log path: /public/home/zenglingsen/01.data/01.ONT_data/02.ONT_test/AW/FAST5_PASS/out/guppy_log chunk size: 2000 chunks per runner: 512 max queued reads: 2000 num basecallers: 4 num socket threads: 2 max returned events: 50000 gpu device: cuda:1 kernel path:
runners per device: 4 2023-08-08 10:17:01.475945 [guppy/info] crashpad_handler not supported on this platform. 2023-08-08 10:17:01.478127 [guppy/info] Listening on port ipc:///tmp/3763-2667-29cb-f0a3. 2023-08-08 10:17:01.607464 [guppy/error] CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:232: CUDA_ERROR_INVALID_DEVICE. Error initialising basecall server using port: ipc://auto. Aborting. 2023-08-08 10:17:01.608638 [guppy/message] The basecall server has shut down successfully.
the last one is the GPU information in partition:
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:86:00.0 Off | N/A | | 44% 33C P0 93W / 350W | 0MiB / 24576MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+
I have followed the reasonable steps through discussion like Guppy can't run on GPU with trained model #46, error while running guppy 4.4.1 on GPU mode #6, and so on. But it does not work.
Any advice and comment are greatful. Thank you very much.