search

hasindu2008 / slow5tools

Slow5tools is a toolkit for converting (FAST5 <-> SLOW5), compressing, viewing, indexing and manipulating data in SLOW5 format.
https://hasindu2008.github.io/slow5tools
MIT License
90 stars 6 forks source link

Bad fast5: Attribute Raw/tracked_scaling_scale #106

Closed nickst85 closed 2 weeks ago

nickst85 commented 6 months ago

Hi I got the following error messages: Bad fast5: Attribute Raw/tracked_scalingscale in /groups/chengrp/dRNAseq/157-DEXdRNAseq/157-DEX/20231031_0244_MC-110944_FAQ79114_c4653c1d/fast5/FAQ79114_c4653c1d_d8ce327a_77.fast5 is unexpected. Please report this with an example FAST5 file at 'https://github.com/hasindu2008/slow5tools/issues' for us to investigate.

Bad fast5: Could not iterate over the read groups in the fast5 file /groups/chengrp/dRNAseq/157_-DEXdRNAseq/157-DEX/20231031_0244_MC-110944_FAQ79114_c4653c1d/fast5/FAQ79114_c4653c1d_d8ce327a_77.fast5.

Bad fast5: Could not read contents of the fast5 file '/groups/chengrp/dRNAseq/157_-DEXdRNAseq/157-DEX/20231031_0244_MC-110944_FAQ79114_c4653c1d/fast5/FAQ79114_c4653c1d_d8ce327a_77.fast5

The same errors had other fast5 files. Let me know how I can share the fast5 file as it's too large to be attached.

Thanks in advance

hasindu2008 commented 6 months ago

Could you please confirm if you are using the latest version of slow5tools? I remember that we handled this attribute in a recent version.

nickst85 commented 6 months ago

Thnak you for your prompt response. I am using the following version.

slow5tools 0.6.0

Is it the latest? Which one should I use?

Best,

Nick

From: Hasindu Gamaarachchi @.> Date: Thursday, December 21, 2023 at 12:05 PM To: hasindu2008/slow5tools @.> Cc: Settas, Nikolaos @.>, Author @.> Subject: [EXT] Re: [hasindu2008/slow5tools] Bad fast5: Attribute Raw/tracked_scaling_scale (Issue #106) ATTENTION: External Email! Do not click attachments/links unless sender is known.

Could you please confirm if you are using the latest version of slow5tools? I remember that we handled this attribute in a recent version. — Reply to this email directly, view it on GitHub [github. com], or unsubscribe [github. com]. You

Could you please confirm if you are using the latest version of slow5tools? I remember that we handled this attribute in a recent version.

— Reply to this email directly, view it on GitHub [github.com]https://urldefense.com/v3/__https:/github.com/hasindu2008/slow5tools/issues/106*issuecomment-1866655623__;Iw!!FMDslA!f6tz5PIaDubZwKk1U34Tal1vkjUIaKFcSLuiG1Mk2pno4abW39DWrM76Oyn7HN_M4F9O779pqPeSPE7iQxjPJDguzu9JM3I$, or unsubscribe [github.com]https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A5QGITRPUMSOIUGBL42WNBDYKRT5RAVCNFSM6AAAAABA6WBIASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRWGY2TKNRSGM__;!!FMDslA!f6tz5PIaDubZwKk1U34Tal1vkjUIaKFcSLuiG1Mk2pno4abW39DWrM76Oyn7HN_M4F9O779pqPeSPE7iQxjPJDguliI_H9k$. You are receiving this because you authored the thread.Message ID: @.***>

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

hasindu2008 commented 6 months ago

0.6 is a very old version. Could you use 1.1?

nickst85 commented 6 months ago

Hi,

I was able to download v1.0.0 and it read the files and finished the job successfully. Thanks for yout help, N From: Hasindu Gamaarachchi @.> Date: Thursday, December 21, 2023 at 7:24 PM To: hasindu2008/slow5tools @.> Cc: Settas, Nikolaos @.>, Author @.> Subject: [EXT] Re: [hasindu2008/slow5tools] Bad fast5: Attribute Raw/tracked_scaling_scale (Issue #106) ATTENTION: External Email! Do not click attachments/links unless sender is known.

0. 6 is a very old version. Could you use 1. 1? — Reply to this email directly, view it on GitHub [github. com], or unsubscribe [github. com]. You are receiving this because you authored the thread. Message ID: hasindu2008/slow5tools/issues/106/1867080904@ github. com

0.6 is a very old version. Could you use 1.1?

— Reply to this email directly, view it on GitHub [github.com]https://urldefense.com/v3/__https:/github.com/hasindu2008/slow5tools/issues/106*issuecomment-1867080904__;Iw!!FMDslA!agPwzXppk56MbAgau6tlWaFUsCWL209AV-b7uwMsQ1qkjrZ5Wsy-ZawxKTHfZM4MfU1n9J1xZHUc_Gdvk-pLkzKsMp-fshE$, or unsubscribe [github.com]https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A5QGITXWBWGTKUJWBRU4QRTYKTHJVAVCNFSM6AAAAABA6WBIASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXGA4DAOJQGQ__;!!FMDslA!agPwzXppk56MbAgau6tlWaFUsCWL209AV-b7uwMsQ1qkjrZ5Wsy-ZawxKTHfZM4MfU1n9J1xZHUc_Gdvk-pLkzKsb4ppwA8$. You are receiving this because you authored the thread.Message ID: @.***>

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

rugilemat commented 4 months ago

Hi,

I seem to be having the same issue (slow5tools 0.6.0) - I know it's an old version but the only way I seem to have been able to successfully install slow5tools on our HPC is through conda (conda install bioconda::slow5tools) as I don't have the permissions to use sudo for the HPC. So would it be possible to update the conda version?

Also I don't know if this is at all relevant but I am trying to run it on the fast5 generated from the RNA flow cells for RNA004 library preps.

This is the code I ran: slow5tools f2s ${fast5} -d ${output}

This is how my error is looking:

[fast5_attribute_itr::ERROR] Bad fast5: Attribute Raw/tracked_scaling_scale in /scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all/PAU73469_pass_e828644c_c4f9b990_356.fast5 is unexpected. Please report this with an example FAST5 file at 'https://github.com/hasindu2008/slow5tools/issues' for us to investigate.
[read_fast5::ERROR] Bad fast5: Could not iterate over the read groups in the fast5 file /scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all/PAU73469_pass_e828644c_c4f9b990_356.fast5.
[f2s_child_worker::ERROR] Bad fast5: Could not read contents of the fast5 file '/scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all/PAU73469_pass_e828644c_c4f9b990_356.fast5'.
[f2s_iop] Child process 2829159 exited with status=1.
hasindu2008 commented 4 months ago

Slowtools 1.1.0 is available on conda https://anaconda.org/bioconda/slow5tools. Could you please try that?

rugilemat commented 4 months ago

Thanks for the quick response! It seems when I try to install slow5tools through conda without specifying the version, it ends up being slow5tools 0.6.0.

I tried conda install bioconda::slow5tools=1.1.0 but it seems to be struggling with the package requirements:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: / 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                        

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package libgcc-ng conflicts for:
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libgcc-ng[version='>=12']
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libaec[version='>=1.0.6,<2.0a0'] -> libgcc-ng[version='>=10.3.0|>=11.2.0|>=9.4.0|>=13.2.0|>=12.3.0|>=11.4.0|>=7.5.0']
libzlib[version='>=1.2.13,<1.3.0a0'] -> libgcc-ng[version='>=12']
zlib -> libgcc-ng[version='>=10.3.0|>=12|>=7.5.0|>=7.3.0|>=4.9|>=11.2.0|>=7.2.0']
libgcc-ng[version='>=12']

Package libzlib conflicts for:
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libcurl[version='>=7.83.1,<9.0a0'] -> libzlib[version='>=1.2.11,<1.3.0a0']
libzlib[version='>=1.2.13,<1.3.0a0']
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libzlib[version='>=1.2.12,<1.3.0a0|>=1.2.13,<1.3.0a0']
zlib -> libzlib[version='1.2.11|1.2.11|1.2.11|1.2.12|1.2.12|1.2.12|1.2.12|1.2.12|1.2.13|1.2.13|1.3',build='h36c2ea0_1012|h166bdaf_2|h166bdaf_3|h166bdaf_4|h166bdaf_4|hd590300_5|hd590300_0|h166bdaf_1|h166bdaf_0|h166bdaf_1014|h36c2ea0_1013']

Package libstdcxx-ng conflicts for:
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libstdcxx-ng[version='>=12']
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libaec[version='>=1.0.6,<2.0a0'] -> libstdcxx-ng[version='>=10.3.0|>=9.4.0']
libstdcxx-ng[version='>=12']

Package zlib conflicts for:
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libcurl[version='>=7.87.0,<9.0a0'] -> zlib[version='>=1.2.12,<1.3.0a0|>=1.2.13,<1.3.0a0']
zlibThe following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.31=0
  - feature:|@/linux-64::__glibc==2.31=0
  - hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libgfortran-ng -> __glibc[version='>=2.17']
  - zlib -> libgcc-ng[version='>=10.3.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.31

Note that strict channel priority may have removed packages required for satisfiability.
hiruna72 commented 4 months ago

Hello @rugilemat

Thanks for reporting this. Until we fix the conda issue can you download the executable to your HPC home directory as follows,

cd ${HOME}
wget https://github.com/hasindu2008/slow5tools/releases/download/v1.1.0/slow5tools-v1.1.0-x86_64-linux-binaries.tar.gz
tar -xvf slow5tools-v1.1.0-x86_64-linux-binaries.tar.gz 
./slow5tools-v1.1.0/slow5tools --version

Now you can set a bash variable to the path of slow5tools executable and use it in your HPC script.

#!/bin/bash

SLOW5TOOLS="${HOME}/slow5tools-v1.1.0/slow5tools"
${SLOW5TOOLS} --version
rugilemat commented 4 months ago

Thanks so much for such a quick response! I am trying it out now and will report back if the issue I've seen above persists.

Edit: it looks like it has worked successfully now. I had to install vbz locally too:

[main] real time = 0.000 sec | CPU time = 0.006 sec | peak RAM = 0.002 GB
slow5tools 1.1.0
[list_all_items] Looking for '*.fast5' files in /scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all
[f2s_main] 1237 fast5 files found - took 0.028s
[f2s_main] Just before forking, peak RAM = 0.000 GB
[f2s_iop] 8 proceses will be used.
[f2s_iop] Spawning 8 I/O processes to circumvent HDF hell.
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 152, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_main] Converting 1237 fast5 files took 2002.569s
[f2s_main] Children processes: CPU time = 11280.989 sec | peak RAM = 0.308 GB

[main] cmd: /scratch/users/k19022845/scripts/slow5tools-v1.1.0/slow5tools f2s /scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all -d /scratch/prj/ppn_microglia_mod/directrna/full_exp/007_CTF_MGL_D1/blow5
[main] real time = 2002.600 sec | CPU time = 0.014 sec | peak RAM = 0.005 GB