Error in thread

[osvaldoreisss]

Hi, I'm having different errors when running isaac. My command line is:

../../Isaac3/bin/isaac-align -r /strg02/users/osvaldo/hg38/human_hg38/sorted-reference.xml -b ../libs_isaac3/ -m 256 --base-calls-format fastq-gz --known-indels /data/users/osvaldo/tmo/known_indels/1000G_phase1.indels.hg38.vcf

I tried with different libraries and still having the same problem:

Do you know whats I'm doing wrong?

Thank's

Dynamic exception type: boost::exception_detail::clone_implisaac::common::ThreadingException std::exception::what: Terminating due to failures on other threads : Terminating due to failures on other threads 2016-06-20 12:37:06 [7fae72360700] ERROR: Thread: 121 also caught an exception: 2016-Jun-20 12:37:06: Success: /data/users/osvaldo/tmo/Isaac3/src/c++/lib/build/Build.cpp(761): Throw in function void isaac::build::Build::waitForLoadSlot(boost::unique_lockboost::mutex&, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator&, isaac::common::ScopedMallocBlock&, std::size_t) Dynamic exception type: boost::exception_detail::clone_implisaac::common::ThreadingException std::exception::what: Terminating due to failures on other threads : Terminating due to failures on other threads 2016-06-20 12:37:18 [7fae4671a700] Serializing records done: 5137358 of them for bin BinMetadata(51id ReferencePosition(4:48388960:0f)bs 12097240bl 1578980154ds 0do 0se 2586255rs 2550203f /data/users/osvaldo/tmo/Tismoo-003/align_isaac3/./Temp/bin-00000005-00000051.dat) in 48seconds. 2016-06-20 12:37:18 [7fae4671a700] ERROR: Thread: 54 also caught an exception: 2016-Jun-20 12:37:18: Success: /data/users/osvaldo/tmo/Isaac3/src/c++/lib/build/Build.cpp(929): Throw in function void isaac::build::Build::preemptComputeSlot(boost::unique_lockboost::mutex&, std::size_t, std::size_t, OperationT, unsigned int) [with OperationT = isaac::build::Build::sortBinParallel(std::vector<boost::reference_wrapper >::const_iterator&, std::vector<boost::reference_wrapper >::const_iterator&, std::vector<boost::reference_wrapper >::const_iterator&, std::vector<boost::reference_wrapper >::const_iterator&, std::vector<boost::reference_wrapper >::const_iterator&, isaac::common::ScopedMallocBlock&, std::size_t)::lambda11; std::size_t = long unsigned int] Dynamic exception type: boost::exception_detail::clone_implisaac::common::ThreadingException std::exception::what: Terminating due to failures on other threads : Terminating due to failures on other threads 2016-06-20 12:37:18 [7fae7916b700] ERROR: Thread: 132 also caught an exception: 2016-Jun-20 12:37:18: Success: /data/users/osvaldo/tmo/Isaac3/src/c++/lib/build/Build.cpp(761): Throw in function void isaac::build::Build::waitForLoadSlot(boost::unique_lockboost::mutex&, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator&, isaac::common::ScopedMallocBlock&, std::size_t) Dynamic exception type: boost::exception_detail::clone_implisaac::common::ThreadingException std::exception::what: Terminating due to failures on other threads : Terminating due to failures on other threads 2016-06-20 12:37:24 [7fae358ff700] Reading alignment records done from BinMetadata(53id ReferencePosition(4:72583440:0f)bs 12097240bl 1647923336ds 0do 0se 2694650rs 2667212f /data/users/osvaldo/tmo/Tismoo-003/align_isaac3/./Temp/bin-00000005-00000053.dat) 2016-06-20 12:37:24 [7fae358ff700] Collecting gaps. 2016-06-20 12:37:25 [7fae358ff700] Finalizing gaps. 2016-06-20 12:37:25 [7fae358ff700] Loading unsorted data done in 16970ms 2016-06-20 12:37:25 [7fae358ff700] ERROR: Thread: 27 also caught an exception: 2016-Jun-20 12:37:25: Success: /data/users/osvaldo/tmo/Isaac3/src/c++/lib/build/Build.cpp(929): Throw in function void isaac::build::Build::preemptComputeSlot(boost::unique_lockboost::mutex&, std::size_t, std::size_t, OperationT, unsigned int) [with OperationT = isaac::build::Build::sortBinParallel(std::vector<boost::reference_wrapper >::const_iterator&, std::vector<boost::reference_wrapper >::const_iterator&, std::vector<boost::reference_wrapper >::const_iterator&, std::vector<boost::reference_wrapper >::const_iterator&, std::vector<boost::reference_wrapper >::const_iterator&, isaac::common::ScopedMallocBlock&, std::size_t)::lambda9; std::size_t = long unsigned int] Dynamic exception type: boost::exception_detail::clone_implisaac::common::ThreadingException std::exception::what: Terminating due to failures on other threads : Terminating due to failures on other threads 2016-06-20 12:37:25 [7fae57f36700] ERROR: Thread: 82 also caught an exception: 2016-Jun-20 12:37:25: Success: /data/users/osvaldo/tmo/Isaac3/src/c++/lib/build/Build.cpp(761): Throw in function void isaac::build::Build::waitForLoadSlot(boost::unique_lockboost::mutex&, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator&, isaac::common::ScopedMallocBlock&, std::size_t) Dynamic exception type: boost::exception_detail::clone_implisaac::common::ThreadingException std::exception::what: Terminating due to failures on other threads : Terminating due to failures on other threads 2016-06-20 12:37:25 [7fae6e75a700] ERROR: Thread: 115 also caught an exception: 2016-Jun-20 12:37:25: Success: /data/users/osvaldo/tmo/Isaac3/src/c++/lib/build/Build.cpp(761): Throw in function void isaac::build::Build::waitForLoadSlot(boost::unique_lockboost::mutex&, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator&, isaac::common::ScopedMallocBlock&, std::size_t) Dynamic exception type: boost::exception_detail::clone_implisaac::common::ThreadingException std::exception::what: Terminating due to failures on other threads : Terminating due to failures on other threads 2016-06-20 12:37:25 [7fae8317b700] ERROR: Thread: 148 also caught an exception: 2016-Jun-20 12:37:25: Success: /data/users/osvaldo/tmo/Isaac3/src/c++/lib/build/Build.cpp(761): Throw in function void isaac::build::Build::waitForLoadSlot(boost::unique_lockboost::mutex&, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator, std::vector<boost::reference_wrapper >::const_iterator&, isaac::common::ScopedMallocBlock&, std::size_t) Dynamic exception type: boost::exception_detail::clone_implisaac::common::ThreadingException std::exception::what: Terminating due to failures on other threads : Terminating due to failures on other threads

[rpetrovski]

What is the first ERROR message? Can you post the whole log file please?

[dearxxj]

I got the same error. Is it because that I compiled issac3 with gcc-4.9.4? I specified the three parameter: -m200 -j64 --use-bases-mask y101,y101

Below is the first ERROR message, can you find out the problem? Thanks!

ERROR: Thread: 23 caught an exception first: 2017-Mar-05 18:56:32: Input/output error: /hpc/home/jux111/src/isaac3/src/c++/lib/alignment/matchSelector/FragmentBinner.cpp(198): Throw in function void isaac::alignment::matchSelector::FragmentBinner::reopenBin(const isaac::alignment::BinMetadata&, std::size_t) Dynamic exception type: boost::exception_detail::clone_implstd::exception::what: Failed to unlink /projects/YueLab/Jie/wgs-total/raw-data/NA12878/./Temp/bin-00000030-00000119.dat : Failed to unlink /projects/YueLab/Jie/wgs-total/raw-data/NA12878/./Temp/bin-00000030-00000119.dat 2017-03-05 18:56:32 [7f41a3a4a700]

[rpetrovski]

Like I've mentioned before the original description has no information on the initial error, however from the fact that it occurs in bam generation and yours does during the alignment, it is easy to conclude that they are unrelated.

In your case there is "Input/output error" (errno EIO) when iSAAC is trying to make sure the old temporary files are not in the way. If the problem is reproducible you should look into the storage. For example the network instability might result in similar effects when the Temp is on a network storage. You might want to try running with Temp directed elsewhere (local disk if it fails on the network or network if it fails on the local disk) to better understand the root cause.

Roman.

[dearxxj]

Hi. I tried to put the output folder in a different place from where the fastq files are located. I also tried to use only one thread (-j 1). The same error was thrown out. I am not sure how to put the Temp folder to local disk or network, since the data are on our clusters and isaac3 was run with SLURM scripts.

[rpetrovski]

The location of Temp can be controlled with -t argument. However if you don't have a local disk on the node, playing with this will unlikely be a useful exercise.

The error you are experiencing is due to an attempt to delete a file which normally should not exist in the first place. iSAAC expects the unlink call to set errno to ENOENT in this case and fails otherwise.

We can check if it is a peculiarity of your system to set the EIO instead. For this simply comment out the line 198 in /hpc/home/jux111/src/isaac3/src/c++/lib/alignment/matchSelector/FragmentBinner.cpp and recompile iSAAC.

Let me know what you find.

Roman.

[dearxxj]

After I commented out that line, the error was eliminated and isaac3 works perfectly! Thanks a lot! If this is a common problem, I guess you can consider removing that line in future versions. :)

[rpetrovski]

No, this is not a common problem. Would you be able to supply a few details about your operating system and storage?

Thank you.

[dearxxj]

System: Linux 2.6.32-642.6.1.el6.x86_64 #1 SMP Thu Aug 25 12:42:19 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Storage shows there is 20T available.

[rpetrovski]

Assuming you are using network storage, can you please find out more details about the storage: Isilon, EMC and such: (https://en.wikipedia.org/wiki/List_of_NAS_manufacturers)