search

nanoporetech / megalodon

Megalodon is a research command line tool to extract high accuracy modified base and sequence variant calls from raw nanopore reads by anchoring the information rich basecalling neural network output to a reference genome/transriptome.
Other
197 stars 30 forks source link

segfault in remora/model_util.py load_onnx_model() #241

Closed benbfly closed 2 years ago

benbfly commented 2 years ago

Using: Megalodon/2.4.1 guppy 6.0.1

I am trying to install and run megalodon 2.4.1 GPU mode. It works fine with the old modbases model, but I am running into problems with Remora. It segfaults inside of remora/model_util.py load_onnx_model(). I'm guessing this is an incompatibility with either cuda libraries, or maybe the Pandas/numexpr warning at the top (remora/model_util.py does use Pandas).

command: megalodon ./megtemp.CPU16.RAM127000.1643019236/1-1 --output-directory ./megout.CPU16.RAM127000.1643019236/1-1 --guppy-params "--use_tcp 34487" --remora-modified-bases dna_r9.4.1_e8 fast 0.0.0 5hmc_5mc CG 0 --guppy-config dna_r9.4.1_450bps_fast.cfg --guppy-server-path /usr/local/hurcs/guppy/6.0.1/bin/guppy_basecall_server --outputs basecalls mappings mod_mappings mods per_read_mods --reference genome.fa --processes 16 --overwrite --devices 0 --suppress-progress-bars --suppress-queues-status

stderr: [12:14:14] Loading guppy basecalling backend [12:14:16] CRF models are not fully supported. /usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/pandas/compat/_optional.py:138: UserWarning: Pandas requires version '2.7.0' or newer of 'numexpr' (version '2.6.9' currently installed). warnings.warn(msg, UserWarning) Traceback (most recent call last): File "/usr/local/hurcs/megalodon/2.4.1/bin/megalodon", line 8, in sys.exit(_main()) File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/megalodon/main.py", line 754, in _main megalodon._main(args) File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/megalodon/megalodon.py", line 1807, in _main args.remora_modified_bases, File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/megalodon/backends.py", line 596, in init remora_model_spec=remora_model_spec, File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/megalodon/backends.py", line 1170, in pyguppy_load_settings remora_model_spec=remora_model_spec, File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/megalodon/backends.py", line 1130, in pyguppy_set_model_attributes quiet=True, File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/remora/model_util.py", line 474, in load_model return load_onnx_model(path, device, quiet=quiet) File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/remora/model_util.py", line 298, in load_onnx_model model_filename, providers=providers, provider_options=provider_options File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 335, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "/usr/local/hurcs/megalodon/2.4.1/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 368, in _create_inference_session sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model) RuntimeError: /onnxruntime_src/onnxruntime/core/platform/posix/env.cc:183 onnxruntime::{anonymous}::PosixThread::PosixThread(const char, int, unsigned int ()(int, Eigen::ThreadPoolInterface), Eigen::ThreadPoolInterface, const onnxruntime::ThreadOptions&) pthread_setaffinity_np failed, error code: 0 error msg:

/var/spool/slurmd/job1134068/slurm_script: line 7: 36960 Segmentation fault

marcus1487 commented 2 years ago

This seems quite similar to another onnx related issue another remora user has encountered. Could you try the python commands in this onnx issue (https://github.com/microsoft/onnxruntime/issues/10166) and report the results?

benbfly commented 2 years ago

Mine produces the same output and stalls: /usr/local/hurcs/megalodon/2.4.1/bin/python3.7 snippet.py 2022-01-24 23:34:17.075188907 [I:onnxruntime:, inference_session.cc:273 operator()] Flush-to-zero and denormal-as-zero are off 2022-01-24 23:34:17.075230285 [I:onnxruntime:, inference_session.cc:280 ConstructorCommon] Creating and using per session threadpools since use_per_sessionthreads is true

Debian v.10 (buster)

marcus1487 commented 2 years ago

This issue should be resolved with the latest Remora release (0.1.2). Please re-open if this issue persists.