search

nanoporetech / ont_fast5_api

Oxford Nanopore Technologies fast5 API software
Other
144 stars 28 forks source link

Error when running compress_fast5 but seems to complete #46

Closed callumparr closed 3 years ago

callumparr commented 3 years ago

I recently run the compress_fast5 with the --sanitize for the sole benefit of removing MinKNOW basecall groups before I proceed to do anything. It seems to run but it will output lots of errors around halfway through. I've check the input and output files and it seems like its worked. Firstly it outputs the same no. of files, viewing in HDFviewer I can the Analyses group has been stripped, and file size is smaller with an equivalence. As they were initially vbz compressed, setting --compression vbz does not seem to alter anything relating to compression.

Can I just ignore these? 

❯ echo $HDF5_PLUGIN_PATH
/usr/local/hdf5/lib/plugin
❯ compress_fast5 --input_path /media/minion/4DC6-8E44/PoreC_A549/PoreC_no_SPRI/Run2/20201005_0901_MN22588_FAN27298_a663793b --save_path /media/minion/4DC6-8E44/PoreC_A549/PoreC_no_SPRI/Run2/20201005_0901_MN22588_FAN27298_a663793b/compressed_fast5_sanitize --compression vbz --recursive --threads 12 --sanitize
ERROR:root:Unable to open file (file signature not found)##############                                                                      | 45% ETA:  0:19:27

then this error repeated a lot 

Traceback (most recent call last):
  File "/home/minion/miniconda3/lib/python3.6/site-packages/ont_fast5_api/conversion_tools/compress_fast5.py", line 55, in compress_file
    if is_multi_read(input_file):
  File "/home/minion/miniconda3/lib/python3.6/site-packages/ont_fast5_api/fast5_interface.py", line 40, in is_multi_read
    with MultiFast5File(filepath, mode='r') as f5_file:
  File "/home/minion/miniconda3/lib/python3.6/site-packages/ont_fast5_api/multi_fast5.py", line 13, in __init__
    self.handle = h5py.File(self.filename, self.mode)
  File "/home/minion/miniconda3/lib/python3.6/site-packages/h5py/_hl/files.py", line 312, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
  File "/home/minion/miniconda3/lib/python3.6/site-packages/h5py/_hl/files.py", line 142, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 78, in h5py.h5f.open
OSError: Unable to open file (file signature not found)
ERROR:root:Unable to open file (file signature not found)

OSError: Unable to open file (file signature not found)
| 1030 of 1030|##############################################################################################################################|100% Time: 0:18:43
fbrennen commented 3 years ago

Hi @callumparr -- thanks for letting us know. We'll have a look.

fbrennen commented 3 years ago

Hi @callumparr -- it looks like this error usually occurs when a file is corrupt or incorrect. Are you using files that you've rescued using MinKNOW's read recovery script, or that you've manually converted from tmp to fast5?

If you can isolate this to a particular file and upload it then we can have a look.

callumparr commented 3 years ago

Hi @callumparr -- it looks like this error usually occurs when a file is corrupt or incorrect. Are you using files that you've rescued using MinKNOW's read recovery script, or that you've manually converted from tmp to fast5?

If you can isolate this to a particular file and upload it then we can have a look.

there was nothing particularly weird about them. all reads passed to the basecalled folder once run was completed. There was several runs for this particular sample and original storage may have be different and i ended up collating them all in once SSD drive. I've been seeing some input/ouput error messages when moving things around. I will dig one multi_fast5 file out and see if I can upload somewhere.