search

nanoporetech / ont_fast5_api

Oxford Nanopore Technologies fast5 API software
Other
145 stars 28 forks source link

Unable to cobvert multi_to_single_fast5 #10

Closed icedevil2001 closed 5 years ago

icedevil2001 commented 5 years ago

Hello We are unable to convert multi fast5s to single fast5 and vice versa. Are there any other checks I could do to debug this issue?

Many thanks!

Error

ERROR:ont_fast5_api.conversion_tools.multi_to_single_fast5:Can't decrement id ref count (file write failed: time = Tue Feb 26 15:04:09 2019 ,filename = 'single2multi_f5_test/1/0033ec25-2ecc-4e76-b040-86fbad852d78.fast5', file descriptor = 6, errno = 5, error message = 'Input/output error', buf = 0x27daf18, total write size = 3040, bytes this sub-write = 3040, bytes actually written = 18446744073709551615, offset = 6336) Failed to copy read '0033ec25-2ecc-4e76-b040-86fbad852d78' from data/FAST5/multi/subset/FAKxxxx_2cc046c36949ccb5fde1bb37b6ed9f35f5d3116a_95.fast5 ERROR:ont_fast5_api.conversion_tools.multi_to_single_fast5:Can't decrement id ref count (unable to close file, errno = 5, error message = 'Input/output error')

Conda env:

# packages in environment at xxxx/miniconda3/envs/fast5_api:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.6             h14c3975_1002    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py36_1000    conda-forge
h5py                      2.9.0                    pypi_0    pypi
libffi                    3.2.1             hf484d3e_1005    conda-forge
libgcc-ng                 7.3.0                hdf63c60_0    conda-forge
libstdcxx-ng              7.3.0                hdf63c60_0    conda-forge
ncurses                   5.9                          10    conda-forge
numpy                     1.16.2                   pypi_0    pypi
ont-fast5-api             1.2.0                    pypi_0    pypi
openssl                   1.0.2q               h14c3975_0    conda-forge
pip                       19.0.3                   py36_0    conda-forge
progressbar33             2.4                      pypi_0    pypi
python                    3.6.0                         2    conda-forge
readline                  6.2                           0    conda-forge
setuptools                40.8.0                   py36_0    conda-forge
six                       1.12.0                   pypi_0    pypi
sqlite                    3.13.0                        1    conda-forge
tk                        8.5.19                        2    conda-forge
wheel                     0.33.1                   py36_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge
fbrennen commented 5 years ago

Hi @icedevil2001 -- are you converting a particularly large number of files? Did you see the same error when you ran multi_to_single_fast5 a second time?

icedevil2001 commented 5 years ago

Hi @fbrennen, there is probably only 10 multi Fast5 files batched into 8000 reads each - I've tried this on my local machine which appears to work correctly however on the main server we still get the above error

fbrennen commented 5 years ago

Great, thanks. Did you see the issue when running the same command a second time? I'm looking at that unable to close file bit and wondering if it was a filesystem issue that might have gone away later.

icedevil2001 commented 5 years ago

I ran it several times with different output directories and had the same issue. However, downgrading h5py from 2.9 to 2.7 resolved my issue. Thank you

fbrennen commented 5 years ago

Ooo, really! Thanks for that -- we'll go see if h5py version makes a difference.

fbrennen commented 5 years ago

To follow up, I tried using h5py 2.9.0 and it worked fine, so I'm still at a loss to explain why this happened. :(