search

nanoporetech / bonito

A PyTorch Basecaller for Oxford Nanopore Reads
https://nanoporetech.com/
Other
395 stars 121 forks source link

Koi beam search decode error #227

Closed esteinig closed 2 years ago

esteinig commented 2 years ago

I'm running into an error using any of the models:

Traceback (most recent call last):
  File "/home/esteinig/miniconda3/envs/bonito/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/esteinig/src/bonito/bonito/multiprocessing.py", line 110, in run
    for item in self.iterator:
  File "/home/esteinig/src/bonito/bonito/crf/basecall.py", line 69, in <genexpr>
    (read, compute_scores(model, batch, reverse=reverse)) for read, batch in batches
  File "/home/esteinig/src/bonito/bonito/crf/basecall.py", line 35, in compute_scores
    sequence, qstring, moves = beam_search(
  File "/home/esteinig/miniconda3/envs/bonito/lib/python3.9/site-packages/koi/decode.py", line 13, in beam_search
    raise TypeError('Expected fp16 but received %s' % scores.dtype)
TypeError: Expected fp16 but received torch.float32

Installation into fresh conda environment with Python 3.9 (this is on a GPU node of our cluster, with CUDA 11.1 module loaded) and either the repository head commit fd5bf56 or v0.5.0

python -m pip install -f https://download.pytorch.org/whl/torch_stable.html --no-cache-dir -r ~/src/bonito/requirements-cuda111.txt
export CUDA_VERSION=111
python -m pip install -e ~/src/bonito

CUDA config:

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  On   | 00000000:83:00.0 Off |                    0 |
| N/A   23C    P0    23W / 250W |      0MiB / 12198MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Dependencies:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             4.5                       1_gnu  
ca-certificates           2021.10.26           h06a4308_2  
certifi                   2021.10.8        py39h06a4308_2  
cffi                      1.15.0                   pypi_0    pypi
chardet                   4.0.0                    pypi_0    pypi
cupy-cuda111              9.6.0                    pypi_0    pypi
fast-ctc-decode           0.3.0                    pypi_0    pypi
fastrlock                 0.8                      pypi_0    pypi
flatbuffers               2.0                      pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
idna                      2.10                     pypi_0    pypi
joblib                    1.1.0                    pypi_0    pypi
koi-cuda111               0.0.5                    pypi_0    pypi
ld_impl_linux-64          2.35.1               h7274673_9  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.3.0               h5101ec6_17  
libgomp                   9.3.0               h5101ec6_17  
libstdcxx-ng              9.3.0               hd4cf53a_17  
mappy                     2.23                     pypi_0    pypi
ncurses                   6.3                  h7f8727e_2  
numpy                     1.19.5                   pypi_0    pypi
onnx                      1.10.2                   pypi_0    pypi
onnxruntime               1.10.0                   pypi_0    pypi
ont-bonito-cuda111        0.5.0                     dev_0    <develop>
ont-fast5-api             3.3.0                    pypi_0    pypi
ont-remora                0.1.1                    pypi_0    pypi
ont-seqdist-cuda111       0.0.4                    pypi_0    pypi
openssl                   1.1.1m               h7f8727e_0  
packaging                 21.3                     pypi_0    pypi
pandas                    1.1.5                    pypi_0    pypi
parasail                  1.2                      pypi_0    pypi
pip                       21.2.4           py39h06a4308_0  
progressbar33             2.4                      pypi_0    pypi
protobuf                  3.19.3                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pyparsing                 3.0.6                    pypi_0    pypi
pysam                     0.18.0                   pypi_0    pypi
python                    3.9.7                h12debd9_1  
python-dateutil           2.8.2                    pypi_0    pypi
pytz                      2021.3                   pypi_0    pypi
readline                  8.1.2                h7f8727e_1  
requests                  2.25.1                   pypi_0    pypi
scikit-learn              1.0.2                    pypi_0    pypi
scipy                     1.5.4                    pypi_0    pypi
setuptools                58.0.4           py39h06a4308_0  
six                       1.16.0                   pypi_0    pypi
sqlite                    3.37.0               hc218d9a_0  
tabulate                  0.8.9                    pypi_0    pypi
thop                      0.0.31-2005241907          pypi_0    pypi
threadpoolctl             3.0.0                    pypi_0    pypi
tk                        8.6.11               h1ccaba5_0  
toml                      0.10.2                   pypi_0    pypi
torch                     1.10.0+cu111             pypi_0    pypi
tqdm                      4.31.1                   pypi_0    pypi
typing-extensions         4.0.1                    pypi_0    pypi
tzdata                    2021e                hda174b7_0  
urllib3                   1.26.8                   pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
xz                        5.2.5                h7b6447c_0  
zlib                      1.2.11               h7f8727e_4
esteinig commented 2 years ago

Ah my apologies looks like this is a known error https://github.com/nanoporetech/bonito/issues/208 - closing issue as it seems like it cannot be resolved