search

Illumina / Isaac4

Isaac aligner version 4
Other
18 stars 3 forks source link

isaac4 for mouse #1

Closed xiexiaowei closed 6 years ago

xiexiaowei commented 6 years ago

Hi,

I successfully installed isaac4 in my linux server, and finished the mapping for human WGS samples quickly! However, when I used isaac4 to map mouse WGS samples. It failed, the error is like this:

2018-01-26 06:17:49 [7f545486a700] ERROR: Thread: 12 caught an exception first: 2018-Jan-26 06:17:49: Success: /public1/users/xxw/software/isaa4/Isaac4-Isaac-04.17.06.15/src/c++/lib/build/Build.cpp(638): Throw in function bool isaac::build::Build::handleBinAllocationFailure(bool, const isaac::alignment::BinMetadata&, const ExceptionType&, const ExceptionDataT&) [with ExceptionType = std::bad_alloc; ExceptionDataT = long unsigned int] Dynamic exception type: boost::exception_detail::clone_impl std::exception::what: ERROR: Failing due to: BinMetadata(29415id ReferencePosition(1:98660000:f)bs 10000bl 13051034234ds 0do 0se 19543367rs 19972219f /public1/users/xxw/isaac/,mouse/ABE_mouse/mDMD-2/./Temp/bin-00000002-094358955.dat) blocking everything with std::bad_alloc : std::bad_alloc Error data: 17317002114 : ERROR: Failing due to: BinMetadata(29415id ReferencePosition(1:98660000:f)bs 10000bl 13051034234ds 0do 0se 19543367rs 19972219f /public1/users/xxw/isaac/,mouse/ABE_mouse/mDMD-2/./Temp/bin-00000002-094358955.dat) blocking everything with std::bad_alloc : std::bad_alloc Error data: 17317002114 .......... : Terminating due to failures on other threads 2018-01-26 06:17:49 [7f553aaeb740] WARNING: rethrowing a thread exception 2018-01-26 06:17:49 [7f553aaeb740] md5 checksum for /public1/users/xxw/isaac/,mouse/ABE_mouse/mDMD-2/out_dir/Projects/default/default/sorted.bam:fb8ffdeeffb283b7cabad84b4296ce94 Error: 2018-Jan-26 06:17:49: Success: /public1/users/xxw/software/isaa4/Isaac4-Isaac-04.17.06.15/src/c++/lib/build/Build.cpp(638): Throw in function bool isaac::build::Build::handleBinAllocationFailure(bool, const isaac::alignment::BinMetadata&, const ExceptionType&, const ExceptionDataT&) [with ExceptionType = std::bad_alloc; ExceptionDataT = long unsigned int] Dynamic exception type: boost::exception_detail::clone_impl std::exception::what: ERROR: Failing due to: BinMetadata(29415id ReferencePosition(1:98660000:f)bs 10000bl 13051034234ds 0do 0se 19543367rs 19972219f /public1/users/xxw/isaac/,mouse/ABE_mouse/mDMD-2/./Temp/bin-00000002-094358955.dat) blocking everything with std::bad_alloc : std::bad_alloc Error data: 17317002114 : ERROR: Failing due to: BinMetadata(29415id ReferencePosition(1:98660000:f)bs 10000bl 13051034234ds 0do 0se 19543367rs 19972219f /public1/users/xxw/isaac/,mouse/ABE_mouse/mDMD-2/./Temp/bin-00000002-094358955.dat) blocking everything with std::bad_alloc : std::bad_alloc Error data: 17317002114

Looking forward to your reply urgently! Thanks very much from my heart! xiaowei,

rpetrovski commented 6 years ago

Looks like some parts of your genome have more data than Isaac expects. If a bin turns out to be too large, it cannot be loaded for bam generation and then you see an error like this one. Can you please post: a. your isaac-align command line b. and the output of the following: ls -alh /public1/users/xxw/isaac/,mouse/ABE_mouse/mDMD-2/./Temp/*

Also, how much RAM do you have available for your runs?

Roman.

xiexiaowei commented 6 years ago

Dear Roman, a. my isaac-align command line: ../bin/isaac-align -j 25 -r ../mm10.fasta -b ../mDMD-2 -f fastq-gz -m 50 \ --base-quality-cutoff 15 --keep-duplicates yes --variable-read-length yes --realign-gaps no \ --default-adapters AGATCGGAAGAGC,GCTCTTCCGATCT -o ../out_dir b. Because I deleted the /Temp files, for the moment, I can't know the size of Temp file. But I have runned that again. c. How to see the available RAM?

Thanks very much again! xiaowei,

rpetrovski commented 6 years ago

free -m should do the job.

Roman.

xiexiaowei commented 6 years ago

a. du -sh /Temp 263G Temp/ b. free -m total used free shared buff/cache available Mem: 161083 53134 435 25 107513 107103 Swap: 163743 1434 162309

It's so strange. The human WGS samples is larrger than the mouse, but human samples completed the mapping.

rpetrovski commented 6 years ago

Hi, could you please do ls - alh Temp

What is the expected coverage of your mouse run?

R

xiexiaowei commented 6 years ago

Dear Roman， running ls -alh Temp, the out is very long because of so much bin..-dat files.

-rw-rw-r-- 1 xxw xxw 38M Jan 26 12:23 AlignerState.txt -rw-rw-r-- 1 xxw xxw 15G Jan 26 12:23 bin-00000000-000000000.dat -rw-rw-r-- 1 xxw xxw 334M Jan 26 12:23 bin-00000001-000000000.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000001-006819581.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000001-013639162.dat -rw-rw-r-- 1 xxw xxw 601M Jan 26 12:23 bin-00000001-020458743.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000001-027278324.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000001-034097905.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000001-040917486.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000001-047737067.dat -rw-rw-r-- 1 xxw xxw 572M Jan 26 12:23 bin-00000001-054556648.dat -rw-rw-r-- 1 xxw xxw 579M Jan 26 12:23 bin-00000001-061376229.dat -rw-rw-r-- 1 xxw xxw 579M Jan 26 12:23 bin-00000001-068195810.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000001-075015391.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000001-081834972.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000001-088654553.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000001-095474134.dat -rw-rw-r-- 1 xxw xxw 580M Jan 26 12:23 bin-00000001-102293715.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000001-109113296.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000001-115932877.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000001-122752458.dat -rw-rw-r-- 1 xxw xxw 645M Jan 26 12:23 bin-00000001-129572039.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000002-005696627.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000002-012516208.dat -rw-rw-r-- 1 xxw xxw 597M Jan 26 12:23 bin-00000002-019335789.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000002-026155370.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000002-032974951.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000002-039794532.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000002-046614113.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000002-053433694.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000002-060253275.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000002-067072856.dat -rw-rw-r-- 1 xxw xxw 596M Jan 26 12:23 bin-00000002-073892437.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000002-080712018.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000002-087531599.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000002-094351180.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000002-101170761.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000002-107990342.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000002-114809923.dat -rw-rw-r-- 1 xxw xxw 2.1G Jan 26 12:23 bin-00000002-121629504.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000003-006366542.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000003-013186123.dat -rw-rw-r-- 1 xxw xxw 598M Jan 26 12:23 bin-00000003-020005704.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000003-026825285.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000003-033644866.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000003-040464447.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000003-047284028.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000003-054103609.dat -rw-rw-r-- 1 xxw xxw 649M Jan 26 12:23 bin-00000003-060923190.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000003-067742771.dat -rw-rw-r-- 1 xxw xxw 660M Jan 26 12:23 bin-00000003-074562352.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000003-081381933.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000003-088201514.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000003-095021095.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000003-101840676.dat -rw-rw-r-- 1 xxw xxw 596M Jan 26 12:23 bin-00000003-108660257.dat -rw-rw-r-- 1 xxw xxw 395M Jan 26 12:23 bin-00000003-115479838.dat -rw-rw-r-- 1 xxw xxw 532M Jan 26 12:23 bin-00000004-002170397.dat -rw-rw-r-- 1 xxw xxw 626M Jan 26 12:23 bin-00000004-008989978.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000004-015809559.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000004-022629140.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000004-029448721.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000004-036268302.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000004-043087883.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000004-049907464.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000004-056727045.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000004-063546626.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000004-070366207.dat -rw-rw-r-- 1 xxw xxw 610M Jan 26 12:23 bin-00000004-077185788.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000004-084005369.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000004-090824950.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000004-097644531.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000004-104464112.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000004-111283693.dat -rw-rw-r-- 1 xxw xxw 342M Jan 26 12:23 bin-00000004-118103274.dat -rw-rw-r-- 1 xxw xxw 598M Jan 26 12:23 bin-00000005-004501216.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000005-011320797.dat -rw-rw-r-- 1 xxw xxw 1.6G Jan 26 12:23 bin-00000005-018140378.dat -rw-rw-r-- 1 xxw xxw 596M Jan 26 12:23 bin-00000005-024959959.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000005-031779540.dat -rw-rw-r-- 1 xxw xxw 631M Jan 26 12:23 bin-00000005-038599121.dat -rw-rw-r-- 1 xxw xxw 594M Jan 26 12:23 bin-00000005-045418702.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000005-052238283.dat -rw-rw-r-- 1 xxw xxw 594M Jan 26 12:23 bin-00000005-059057864.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000005-065877445.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000005-072697026.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000005-079516607.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000005-086336188.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000005-093155769.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000005-099975350.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000005-106794931.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000005-113614512.dat -rw-rw-r-- 1 xxw xxw 370M Jan 26 12:23 bin-00000005-120434093.dat -rw-rw-r-- 1 xxw xxw 526M Jan 26 12:23 bin-00000006-002351430.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000006-009171011.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000006-015990592.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000006-022810173.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000006-029629754.dat -rw-rw-r-- 1 xxw xxw 580M Jan 26 12:23 bin-00000006-036449335.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000006-043268916.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000006-050088497.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000006-056908078.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000006-063727659.dat -rw-rw-r-- 1 xxw xxw 655M Jan 26 12:23 bin-00000006-070547240.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000006-077366821.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000006-084186402.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000006-091005983.dat -rw-rw-r-- 1 xxw xxw 529M Jan 26 12:23 bin-00000006-097825564.dat -rw-rw-r-- 1 xxw xxw 444M Jan 26 12:23 bin-00000007-000601460.dat -rw-rw-r-- 1 xxw xxw 602M Jan 26 12:23 bin-00000007-007421041.dat -rw-rw-r-- 1 xxw xxw 594M Jan 26 12:23 bin-00000007-014240622.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000007-021060203.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000007-027879784.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000007-034699365.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000007-041518946.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000007-048338527.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000007-055158108.dat -rw-rw-r-- 1 xxw xxw 578M Jan 26 12:23 bin-00000007-061977689.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000007-068797270.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000007-075616851.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000007-082436432.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000007-089256013.dat -rw-rw-r-- 1 xxw xxw 363M Jan 26 12:23 bin-00000007-096075594.dat -rw-rw-r-- 1 xxw xxw 602M Jan 26 12:23 bin-00000008-004687407.dat -rw-rw-r-- 1 xxw xxw 640M Jan 26 12:23 bin-00000008-011506988.dat -rw-rw-r-- 1 xxw xxw 607M Jan 26 12:23 bin-00000008-018326569.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000008-025146150.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000008-031965731.dat -rw-rw-r-- 1 xxw xxw 660M Jan 26 12:23 bin-00000008-038785312.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000008-045604893.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000008-052424474.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000008-059244055.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000008-066063636.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000008-072883217.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000008-079702798.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000008-086522379.dat -rw-rw-r-- 1 xxw xxw 371M Jan 26 12:23 bin-00000008-093341960.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000009-005174270.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000009-011993851.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000009-018813432.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000009-025633013.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000009-032452594.dat -rw-rw-r-- 1 xxw xxw 597M Jan 26 12:23 bin-00000009-039272175.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000009-046091756.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000009-052911337.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000009-059730918.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000009-066550499.dat -rw-rw-r-- 1 xxw xxw 576M Jan 26 12:23 bin-00000009-073370080.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000009-080189661.dat -rw-rw-r-- 1 xxw xxw 318M Jan 26 12:23 bin-00000009-087009242.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000010-003126184.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000010-009945765.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000010-016765346.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000010-023584927.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000010-030404508.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000010-037224089.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000010-044043670.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000010-050863251.dat -rw-rw-r-- 1 xxw xxw 354M Jan 26 12:23 bin-00000010-057682832.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000011-003070847.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000011-009890428.dat -rw-rw-r-- 1 xxw xxw 574M Jan 26 12:23 bin-00000011-016710009.dat -rw-rw-r-- 1 xxw xxw 690M Jan 26 12:23 bin-00000011-023529590.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000011-030349171.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000011-037168752.dat -rw-rw-r-- 1 xxw xxw 579M Jan 26 12:23 bin-00000011-043988333.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000011-050807914.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000011-057627495.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000011-064447076.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000011-071266657.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000011-078086238.dat -rw-rw-r-- 1 xxw xxw 857M Jan 26 12:23 bin-00000011-084905819.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000011-091725400.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000011-098544981.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000011-105364562.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000011-112184143.dat -rw-rw-r-- 1 xxw xxw 580M Jan 26 12:23 bin-00000011-119003724.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000011-125823305.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000011-132642886.dat -rw-rw-r-- 1 xxw xxw 576M Jan 26 12:23 bin-00000011-139462467.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000011-146282048.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000011-153101629.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000011-159921210.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000011-166740791.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000011-173560372.dat -rw-rw-r-- 1 xxw xxw 596M Jan 26 12:23 bin-00000011-180379953.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000011-187199534.dat -rw-rw-r-- 1 xxw xxw 327M Jan 26 12:23 bin-00000011-194019115.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000012-005366725.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000012-012186306.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000012-019005887.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000012-025825468.dat -rw-rw-r-- 1 xxw xxw 597M Jan 26 12:23 bin-00000012-032645049.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000012-039464630.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000012-046284211.dat -rw-rw-r-- 1 xxw xxw 575M Jan 26 12:23 bin-00000012-053103792.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000012-059923373.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000012-066742954.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000012-073562535.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000012-080382116.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000012-087201697.dat -rw-rw-r-- 1 xxw xxw 16G Jan 26 12:23 bin-00000012-094021278.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000012-100840859.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000012-107660440.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000012-114480021.dat -rw-rw-r-- 1 xxw xxw 596M Jan 26 12:23 bin-00000012-121299602.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000012-128119183.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000012-134938764.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000012-141758345.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000012-148577926.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000012-155397507.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000012-162217088.dat -rw-rw-r-- 1 xxw xxw 549M Jan 26 12:23 bin-00000012-169036669.dat -rw-rw-r-- 1 xxw xxw 492M Jan 26 12:23 bin-00000012-175856250.dat -rw-rw-r-- 1 xxw xxw 392M Jan 26 12:23 bin-00000013-000562607.dat -rw-rw-r-- 1 xxw xxw 600M Jan 26 12:23 bin-00000013-007382188.dat -rw-rw-r-- 1 xxw xxw 580M Jan 26 12:23 bin-00000013-014201769.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000013-021021350.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000013-027840931.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000013-034660512.dat -rw-rw-r-- 1 xxw xxw 578M Jan 26 12:23 bin-00000013-041480093.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000013-048299674.dat -rw-rw-r-- 1 xxw xxw 597M Jan 26 12:23 bin-00000013-055119255.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000013-061938836.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000013-068758417.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000013-075577998.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000013-082397579.dat -rw-rw-r-- 1 xxw xxw 612M Jan 26 12:23 bin-00000013-089217160.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000013-096036741.dat -rw-rw-r-- 1 xxw xxw 606M Jan 26 12:23 bin-00000013-102856322.dat -rw-rw-r-- 1 xxw xxw 574M Jan 26 12:23 bin-00000013-109675903.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000013-116495484.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000013-123315065.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000013-130134646.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000013-136954227.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000013-143773808.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000013-150593389.dat -rw-rw-r-- 1 xxw xxw 363M Jan 26 12:23 bin-00000013-157412970.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000014-004192871.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000014-011012452.dat -rw-rw-r-- 1 xxw xxw 578M Jan 26 12:23 bin-00000014-017832033.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000014-024651614.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000014-031471195.dat -rw-rw-r-- 1 xxw xxw 577M Jan 26 12:23 bin-00000014-038290776.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000014-045110357.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000014-051929938.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000014-058749519.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000014-065569100.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000014-072388681.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000014-079208262.dat -rw-rw-r-- 1 xxw xxw 575M Jan 26 12:23 bin-00000014-086027843.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000014-092847424.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000014-099667005.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000014-106486586.dat -rw-rw-r-- 1 xxw xxw 597M Jan 26 12:23 bin-00000014-113306167.dat -rw-rw-r-- 1 xxw xxw 626M Jan 26 12:23 bin-00000014-120125748.dat -rw-rw-r-- 1 xxw xxw 574M Jan 26 12:23 bin-00000014-126945329.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000014-133764910.dat -rw-rw-r-- 1 xxw xxw 759M Jan 26 12:23 bin-00000014-140584491.dat -rw-rw-r-- 1 xxw xxw 657M Jan 26 12:23 bin-00000014-147404072.dat -rw-rw-r-- 1 xxw xxw 323M Jan 26 12:23 bin-00000014-154223653.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000015-004535118.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000015-011354699.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000015-018174280.dat -rw-rw-r-- 1 xxw xxw 598M Jan 26 12:23 bin-00000015-024993861.dat -rw-rw-r-- 1 xxw xxw 598M Jan 26 12:23 bin-00000015-031813442.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000015-038633023.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000015-045452604.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000015-052272185.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000015-059091766.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000015-065911347.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000015-072730928.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000015-079550509.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000015-086370090.dat -rw-rw-r-- 1 xxw xxw 745M Jan 26 12:23 bin-00000015-093189671.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000015-100009252.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000015-106828833.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000015-113648414.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000015-120467995.dat -rw-rw-r-- 1 xxw xxw 575M Jan 26 12:23 bin-00000015-127287576.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000015-134107157.dat -rw-rw-r-- 1 xxw xxw 604M Jan 26 12:23 bin-00000015-140926738.dat -rw-rw-r-- 1 xxw xxw 344M Jan 26 12:23 bin-00000015-147746319.dat -rw-rw-r-- 1 xxw xxw 572M Jan 26 12:23 bin-00000016-002731216.dat -rw-rw-r-- 1 xxw xxw 579M Jan 26 12:23 bin-00000016-009550797.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000016-016370378.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000016-023189959.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000016-030009540.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000016-036829121.dat -rw-rw-r-- 1 xxw xxw 618M Jan 26 12:23 bin-00000016-043648702.dat -rw-rw-r-- 1 xxw xxw 594M Jan 26 12:23 bin-00000016-050468283.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000016-057287864.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000016-064107445.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000016-070927026.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000016-077746607.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000016-084566188.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000016-091385769.dat -rw-rw-r-- 1 xxw xxw 1.2G Jan 26 12:23 bin-00000016-098205350.dat -rw-rw-r-- 1 xxw xxw 579M Jan 26 12:23 bin-00000016-105024931.dat -rw-rw-r-- 1 xxw xxw 593M Jan 26 12:23 bin-00000016-111844512.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000016-118664093.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000016-125483674.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000016-132303255.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000016-139122836.dat -rw-rw-r-- 1 xxw xxw 318M Jan 26 12:23 bin-00000016-145942417.dat -rw-rw-r-- 1 xxw xxw 558M Jan 26 12:23 bin-00000017-003025452.dat -rw-rw-r-- 1 xxw xxw 614M Jan 26 12:23 bin-00000017-009845033.dat -rw-rw-r-- 1 xxw xxw 647M Jan 26 12:23 bin-00000017-016664614.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000017-023484195.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000017-030303776.dat -rw-rw-r-- 1 xxw xxw 597M Jan 26 12:23 bin-00000017-037123357.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000017-043942938.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000017-050762519.dat -rw-rw-r-- 1 xxw xxw 601M Jan 26 12:23 bin-00000017-057582100.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000017-064401681.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000017-071221262.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000017-078040843.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000017-084860424.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000017-091680005.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000017-098499586.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000017-105319167.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000017-112138748.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000017-118958329.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000017-125777910.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000017-132597491.dat -rw-rw-r-- 1 xxw xxw 513M Jan 26 12:23 bin-00000017-139417072.dat -rw-rw-r-- 1 xxw xxw 398M Jan 26 12:23 bin-00000018-000795194.dat -rw-rw-r-- 1 xxw xxw 594M Jan 26 12:23 bin-00000018-007614775.dat -rw-rw-r-- 1 xxw xxw 738M Jan 26 12:23 bin-00000018-014434356.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000018-021253937.dat -rw-rw-r-- 1 xxw xxw 576M Jan 26 12:23 bin-00000018-028073518.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000018-034893099.dat -rw-rw-r-- 1 xxw xxw 584M Jan 26 12:23 bin-00000018-041712680.dat -rw-rw-r-- 1 xxw xxw 580M Jan 26 12:23 bin-00000018-048532261.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000018-055351842.dat -rw-rw-r-- 1 xxw xxw 579M Jan 26 12:23 bin-00000018-062171423.dat -rw-rw-r-- 1 xxw xxw 613M Jan 26 12:23 bin-00000018-068991004.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000018-075810585.dat -rw-rw-r-- 1 xxw xxw 591M Jan 26 12:23 bin-00000018-082630166.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000018-089449747.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000018-096269328.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000018-103088909.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000018-109908490.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000018-116728071.dat -rw-rw-r-- 1 xxw xxw 503M Jan 26 12:23 bin-00000018-123547652.dat -rw-rw-r-- 1 xxw xxw 9.9G Jan 26 12:23 bin-00000019-000966020.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000019-007785601.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000019-014605182.dat -rw-rw-r-- 1 xxw xxw 610M Jan 26 12:23 bin-00000019-021424763.dat -rw-rw-r-- 1 xxw xxw 594M Jan 26 12:23 bin-00000019-028244344.dat -rw-rw-r-- 1 xxw xxw 1.3G Jan 26 12:23 bin-00000019-035063925.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000019-041883506.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000019-048703087.dat -rw-rw-r-- 1 xxw xxw 594M Jan 26 12:23 bin-00000019-055522668.dat -rw-rw-r-- 1 xxw xxw 592M Jan 26 12:23 bin-00000019-062342249.dat -rw-rw-r-- 1 xxw xxw 589M Jan 26 12:23 bin-00000019-069161830.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000019-075981411.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000019-082800992.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000019-089620573.dat -rw-rw-r-- 1 xxw xxw 590M Jan 26 12:23 bin-00000019-096440154.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000019-103259735.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000019-110079316.dat -rw-rw-r-- 1 xxw xxw 595M Jan 26 12:23 bin-00000019-116898897.dat -rw-rw-r-- 1 xxw xxw 1.1G Jan 26 12:23 bin-00000019-123718478.dat -rw-rw-r-- 1 xxw xxw 580M Jan 26 12:23 bin-00000021-005926650.dat -rw-rw-r-- 1 xxw xxw 585M Jan 26 12:23 bin-00000021-012746231.dat -rw-rw-r-- 1 xxw xxw 573M Jan 26 12:23 bin-00000021-019565812.dat -rw-rw-r-- 1 xxw xxw 612M Jan 26 12:23 bin-00000021-026385393.dat -rw-rw-r-- 1 xxw xxw 573M Jan 26 12:23 bin-00000021-033204974.dat -rw-rw-r-- 1 xxw xxw 565M Jan 26 12:23 bin-00000021-040024555.dat -rw-rw-r-- 1 xxw xxw 569M Jan 26 12:23 bin-00000021-046844136.dat -rw-rw-r-- 1 xxw xxw 580M Jan 26 12:23 bin-00000021-053663717.dat -rw-rw-r-- 1 xxw xxw 572M Jan 26 12:23 bin-00000021-060483298.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000021-067302879.dat -rw-rw-r-- 1 xxw xxw 597M Jan 26 12:23 bin-00000021-074122460.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000021-080942041.dat -rw-rw-r-- 1 xxw xxw 581M Jan 26 12:23 bin-00000021-087761622.dat -rw-rw-r-- 1 xxw xxw 586M Jan 26 12:23 bin-00000021-094581203.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000021-101400784.dat -rw-rw-r-- 1 xxw xxw 568M Jan 26 12:23 bin-00000021-108220365.dat -rw-rw-r-- 1 xxw xxw 582M Jan 26 12:23 bin-00000021-115039946.dat -rw-rw-r-- 1 xxw xxw 487M Jan 26 12:23 bin-00000021-121859527.dat -rw-rw-r-- 1 xxw xxw 587M Jan 26 12:23 bin-00000021-128679108.dat -rw-rw-r-- 1 xxw xxw 576M Jan 26 12:23 bin-00000021-135498689.dat -rw-rw-r-- 1 xxw xxw 603M Jan 26 12:23 bin-00000021-142318270.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000021-149137851.dat -rw-rw-r-- 1 xxw xxw 583M Jan 26 12:23 bin-00000021-155957432.dat -rw-rw-r-- 1 xxw xxw 588M Jan 26 12:23 bin-00000021-162777013.dat -rw-rw-r-- 1 xxw xxw 60M Jan 26 12:23 bin-00000021-169596594.dat -rw-rw-r-- 1 xxw xxw 3.4M Jan 26 12:23 bin-00000022-005384876.dat -rw-rw-r-- 1 xxw xxw 2.4M Jan 26 12:23 bin-00000022-012204457.dat -rw-rw-r-- 1 xxw xxw 1.7M Jan 26 12:23 bin-00000022-019024038.dat -rw-rw-r-- 1 xxw xxw 1.3M Jan 26 12:23 bin-00000022-025843619.dat -rw-rw-r-- 1 xxw xxw 1.4M Jan 26 12:23 bin-00000022-032663200.dat -rw-rw-r-- 1 xxw xxw 3.0M Jan 26 12:23 bin-00000022-039482781.dat -rw-rw-r-- 1 xxw xxw 1.2M Jan 26 12:23 bin-00000022-046302362.dat -rw-rw-r-- 1 xxw xxw 1.8M Jan 26 12:23 bin-00000022-053121943.dat -rw-rw-r-- 1 xxw xxw 1.1M Jan 26 12:23 bin-00000022-059941524.dat -rw-rw-r-- 1 xxw xxw 588K Jan 26 12:23 bin-00000022-066761105.dat -rw-rw-r-- 1 xxw xxw 1.1M Jan 26 12:23 bin-00000022-073580686.dat

xiaowei,

rpetrovski commented 6 years ago

Xiaowei, I've pushed most recent Isaac4 updates which include

Isaac-04.18.01.18

It addresses a memory allocation issue that is likely to be responsible for your failure.

Please let me know if it solves your problem. Roman.

BenoitFiset commented 6 years ago

Hi Roman,

Trying to aligne whole genome on a 256G of RAM server.... with version of Isaac:

[bfiset@ip20-mp2 bin]$ ./isaac-align -v
2018-08-13 12:48:45     [2b7627b97440]  Forcing LC_ALL to C
2018-08-13 12:48:45     [2b7627b97440]  Version: Isaac-04.18.05.23
2018-08-13 12:48:45     [2b7627b97440]  Genome offset type: j
2018-08-13 12:48:45     [2b7627b97440]  argc: 2 argv: ./isaac-align -v

Command line:

/home/bfiset/lw-project/Isaac4-bin/bin/isaac-align --reference-genome /localscratch/bfiset.45188.0/IsaacIndex_GRCh38_V93/sorted-reference.xml --base-calls /home/bfiset/scratch/WGS/HI.4784-008_4763-006 --memory-limit 250 --base-calls-format fastq --jobs 48 --realign-gaps all --output-directory /localscratch/bfiset.45188.0/HI.4784-008_4763-006_Aligned_01 --temp-directory /localscratch/bfiset.45188.0/Isaac_Temp_DIR

After about 4h of run getting errors of this kind:

2018-08-13 04:07:53     [2aed85c6b700]  ERROR: Thread: 139 caught an exception first: 2018-Aug-13 04:07:53: Success: /home/bfiset/lw-project/Isaac-src/Isaac4-master/src/c++/lib/build/Build.cpp(680): Throw in function boost::shared_ptr<isaac::build::BinData> isaac::build::Build::allocateBin(boost::unique_lock<boost::mutex>&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::const_iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::const_iterator, isaac::common::ScopedMallocBlock&, std::size_t)
Dynamic exception type: boost::exception_detail::clone_impl<isaac::common::ThreadingException>
std::exception::what: Terminating due to failures on other threads
: Terminating due to failures on other threads

Should I be using: Isaac-04.18.01.18 ?

Thanks for the help,

B.

rpetrovski commented 6 years ago

Benoit, I'm away from my computer for about another week. Can you please grep your log for the very first line containing word ERROR and post it.

R.

BenoitFiset commented 6 years ago

Hi Roman,

That was it....

2018-08-13 04:07:53     [2aed85c6b700]  ERROR: Thread: 139 caught an exception first: 2018-Aug-13 04:07:53: Success: /home/bfiset/lw-project/Isaac-src/Isaac4-master/src/c++/lib/build/Build.cpp(680): Throw in function boost::shared_ptr<isaac::build::BinData> isaac::build::Build::allocateBin(boost::unique_lock<boost::mutex>&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::const_iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::const_iterator, isaac::common::ScopedMallocBlock&, std::size_t)
Dynamic exception type: boost::exception_detail::clone_impl<isaac::common::ThreadingException>
std::exception::what: Terminating due to failures on other threads
: Terminating due to failures on other threads

The Ones after are:

2018-08-13 04:07:53     [2aee49b91700]  ERROR: Thread: 73 also caught an exception: 2018-Aug-13 04:07:53: Success: /home/bfiset/lw-project/Isaac-src/Isaac4-master/src/c++/lib/build/Build.cpp(680): Throw in function boost::shared_ptr<isaac::build::BinData> isaac::build::Build::allocateBin(boost::unique_lock<boost::mutex>&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::const_iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::const_iterator, isaac::common::ScopedMallocBlock&, std::size_t)
Dynamic exception type: boost::exception_detail::clone_impl<isaac::common::ThreadingException>
std::exception::what: Terminating due to failures on other threads
: Terminating due to failures on other threads

2018-08-13 04:07:53     [2aed89c8b700]  ERROR: Thread: 126 also caught an exception: 2018-Aug-13 04:07:53: Success: /home/bfiset/lw-project/Isaac-src/Isaac4-master/src/c++/lib/build/Build.cpp(680): Throw in function boost::shared_ptr<isaac::build::BinData> isaac::build::Build::allocateBin(boost::unique_lock<boost::mutex>&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::const_iterator&, std::vector<boost::reference_wrapper<isaac::alignment::BinMetadata> >::const_iterator, isaac::common::ScopedMallocBlock&, std::size_t)
Dynamic exception type: boost::exception_detail::clone_impl<isaac::common::ThreadingException>
std::exception::what: Terminating due to failures on other threads
: Terminating due to failures on other threads
BenoitFiset commented 6 years ago

And the lines before this error:

2018-08-13 04:07:45     [2aee49b91700]  Serializing records done: 8132998 of them for bin BinMetadata(284595id ReferencePosition(21:109250000:f)bs 19100000bl 2352267142ds 0do 0se 3614347rs 3404989f /localscratch
/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-109241045.dat) in 80seconds.
2018-08-13 04:07:45     [2aee49b91700]  Saving 715433258 bytes of sorted data for bin /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-109241045.dat
2018-08-13 04:07:45     [2aee4a797700]  STAT: Before allocating data for BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.45188.0/Isaac_Temp_DI
R/bin-00000022-128340695.dat) 9306054656vm 1463494res
2018-08-13 04:07:50     [2aee49b91700]  Saving 715433258 bytes of sorted data for bin /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-109241045.dat done in 4772ms
2018-08-13 04:07:50     [2aee49b91700]  MERGED targetBinSize_:2582285186 BinMetadata(310083id ReferencePosition(180:0:f)bs 10000bl 70606ds 0do 0se 116rs 96f /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-000001
80-000000000.dat) 7
2018-08-13 04:07:50     [2aed87477700]  Saving 319587161 bytes of sorted data for bin /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-128340695.dat
2018-08-13 04:07:50     [2aee4a797700]  STAT: Before allocating data for BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.45188.0/Isaac_Temp_DI
R/bin-00000022-128340695.dat) 6587940864vm 1288371res
2018-08-13 04:07:52     [2aed87477700]  Saving 319587161 bytes of sorted data for bin /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-128340695.dat done in 2333ms
2018-08-13 04:07:53     [2aed87477700]  MERGED targetBinSize_:2582285186 BinMetadata(310084id ReferencePosition(181:0:f)bs 10000bl 31628712ds 0do 0se 49104rs 45506f /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bi
n-00000180-000000000.dat) 1873
2018-08-13 04:07:53     [2aee4a797700]  STAT: Before allocating data for BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.45188.0/Isaac_Temp_DI
R/bin-00000022-128340695.dat) 5036167168vm 1126318res

The BAM file did start to be generated.

rpetrovski commented 6 years ago

It looks like the thread that caused the failure has not managed to trace it before other threads decided to bail. I expect the original message should still be among those lines. Would you be able to do something like

cat | grep ERROR | grep -v ThreadingException

BenoitFiset commented 6 years ago

Did this:

cat HI_4784-008_4763_006_Isaac_Aligned_01_45188.err | grep ERROR | grep -v Thread

Got only this:

std::exception::what: ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-128340695.dat) blocking everything with std::bad_alloc : std::bad_alloc Error data: 1183078250 
: ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-128340695.dat) blocking everything with std::bad_alloc : std::bad_alloc Error data: 1183078250
BenoitFiset commented 6 years ago

Did a grep for : std::bad_alloc

And got this.... one of them is long unsigned int on Thread 67

2018-08-13 04:07:53     [2aee4a797700]  ERROR: Thread: 67 also caught an exception: 2018-Aug-13 04:07:53: Success: /home/bfiset/lw-project/Isaac-src/Isaac4-master/src/c++/lib/build/Build.cpp(634): Throw in function bool isaac::build::Build::handleBinAllocationFailure(bool, const isaac::alignment::BinMetadata&, const ExceptionType&, const ExceptionDataT&) [with ExceptionType = std::bad_alloc; ExceptionDataT = long unsigned int]

std::exception::what: ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-128340695.dat) blocking everything with std::bad_alloc : std::bad_alloc Error data: 1183078250 
: ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.45188.0/Isaac_Temp_DIR/bin-00000022-128340695.dat) blocking everything with std::bad_alloc : std::bad_alloc Error data: 1183078250
rpetrovski commented 6 years ago

Ok, looks like another one of memory allocation estimation problems. To debug this, I most likely will need your data. At least some of your temp files. Is this an option? Also, I will not be able to work on that until August 24-ish.

If the issue is in the area I am thinking about, the following might help --split-reads no. This of course disables search for structural variants. Would that work for you?

Roman.

BenoitFiset commented 6 years ago

Yish.... Clinical data.... Canada - Border - USA..... pretty not very likely to happen....

I'll help as best as I can.... with your guidance...

SV are part for next steps for this data.... Manta, Strelka, Nirvana..... so yes SV necessary... Would bringing memory argument lower help ? Server with more memory ?

rpetrovski commented 6 years ago

More memory might help. Another trick you can try is running with smaller -m and --stop-at Align. And then running with larger -m and --start-from Align.

You should still try --split-reads no just to see if the failure is related to SV.

Roman

BenoitFiset commented 6 years ago

So to use smaller -m and --stop-at Align and larger -m and --start-from Align I use the dams Temp directory and all other arguments... just that I run the job once with --stop-at Align and then restart job with --start-from Align ?

I have 256g of RAM how much ram you want me to try for each of the 2 steps ?

Full RAM for the --split-reads no ?

rpetrovski commented 6 years ago

Yes.

I'd try 100 and 250.

Yes.

BenoitFiset commented 6 years ago

Hi,

--stop-at and --start-from trick didn't work :(

Same errors...

From this whole error:

2018-08-14 15:38:02     [2acede59f700]  ERROR: Thread: 25 also caught an exception: 2018-Aug-14 15:38:02: Success: /home/bfiset/lw-project/Isaac-src/Isaac4-master/src/c++/lib/build/Build.cpp(634): Throw in funct
ion bool isaac::build::Build::handleBinAllocationFailure(bool, const isaac::alignment::BinMetadata&, const ExceptionType&, const ExceptionDataT&) [with ExceptionType = std::bad_alloc; ExceptionDataT = long unsig
ned int]
std::exception::what: ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.47225.0/Isaac_Temp_DIR/bin-00000022-127588445.dat
) blocking everything with std::bad_alloc : std::bad_alloc Error data: 1183078250 
: ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.47225.0/Isaac_Temp_DIR/bin-00000022-127588445.dat) blocking everythin
g with std::bad_alloc : std::bad_alloc Error data: 1183078250

What does this part mean:

ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)

Is this a position in the file ? Seems to be always the same position. Genome ref file or Seq data file ? Data file bad ?

Running now with --split-reads no for test. Will update.

Thanks.

BenoitFiset commented 6 years ago

Hi,

--split-reads no

I get Error: std::exception::what: unrecognised option '--split-reads'

Would it be: --split-alignments instead ?

Thanks.

rpetrovski commented 6 years ago

Yes. --split-alignments

rpetrovski commented 6 years ago

To answer your previous question, it keeps failing on the same 20kb section of chromosome 22 (assuming your fasta has them in order with 1 being first). I.E. chunk of that chromosome between position 0 and position 20000 has a number of alignments that throw bam generation memory allocation predictor into disarray.

Couple of questions. By default isaac-align runs with --expected-coverage 60.

  1. Is that anywhere near the coverage you are trying to analyse? If not, you might want to try to set the --expected-coverage correctly for the run.
  2. Is your coverage uniform? I.E. I am assuming this is not a targeted sequencing run?

Roman.

BenoitFiset commented 6 years ago

Hi Roman,

1) coverage = 30x so since this paired end I put 60 or 30 ? 2) Uniform coverage

Result: works with using --split-alignments no argument

2018-08-14 21:05:45     [2b964afc0440]  BAM file generated: /localscratch/ ... sorted.bam
2018-08-14 21:05:45     [2b964afc0440]  BAM index generated for /localscratch/ ... sorted.bam
2018-08-14 21:05:45     [2b964afc0440]  Generating Build statistics
2018-08-14 21:05:47     [2b964afc0440]  Generating Build statistics done
2018-08-14 21:05:47     [2b964afc0440]  Generating the BAM files done

I'll wait for your comments about value for coverage before trying to get a 512G server.

B.

rpetrovski commented 6 years ago

I've just checked with the code and it looks like --expected-coverage is not a factor in estimating the memory for splits anymore. It used to be in Isaac-04.18.01.18 so, if you can live without the other fixes that followed, this might be a short term workaround for you.

The subsequent fixes resulted in using the real coverage in the bin (number of aligned bases / bin length). My guess is that you have a pileup at the start of that chromosome, and this pileup brings the local coverage too high, resulting in the overestimation. If that is the case, one option is to use GRC38 with decoys. I've seen this to deal with erroneous structural variants in a number of cases. Is that an option for you?

R

BenoitFiset commented 6 years ago

So option 1 is to use Isaac-04.18.01.18 instead of Isaac-04.18.05.23.. ?

Option 2 is to use Isaac-04.18.05.23 with GRCh38 with decoys ? I'll have to read up on that as I have no clue how this works.

I have one alignment in the same experiment (the Normal) which is about the same size that didn't have any issue with Isaac-04.18.05.23 with the same reference GRCh38 assembly.

When you get back, what would you need from my end to try to debug this issue ?

Thanks.

BenoitFiset commented 6 years ago

Hi Roman,

update, your hunch was right. The alignment passed with Isaac-04.18.01.18

2018-08-20 16:22:18     [2b9a390cf640]  BAM file generated: /localscratch/bfiset.66556.0/HI.4784-008_4763-006_V418_Aligned_01/Projects/default/default/sorted.bam
2018-08-20 16:22:18     [2b9a390cf640]  BAM index generated for /localscratch/bfiset.66556.0/HI.4784-008_4763-006_V418_Aligned_01/Projects/default/default/sorted.bam
2018-08-20 16:22:19     [2b9a390cf640]  Generating Build statistics
2018-08-20 16:22:21     [2b9a390cf640]  Generating Build statistics done
2018-08-20 16:22:21     [2b9a390cf640]  Generating the BAM files done

Waiting for your input / instructions to try and help you out to debug version Isaac-04.18.05.23

B.

rpetrovski commented 6 years ago

The way I understand it, you should have a pileup on the start of the 22nd chromosome counting in the order they appear in you fasta. The version that works uses expected coverage to estimate memory. The one that fails uses the actual coverage in the bin. If you can confirm the pileup and tell me the coverage there, I will try to reconcile the two approaches.

Roman.

BenoitFiset commented 6 years ago

Hi Roman,

excuse my ignorance on pile-ups.... I found a description : "A pile-up is a term used in genetic genealogy to describe multiple shared autosomal DNA segments that are stacked up on top of each other on the same part of the genome."

In this case I'm using GRCh38 V93 (ftp://ftp.ensembl.org/pub/release-93/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz)

You want me to figure out the data in my sequenced data at the start of Chr 22 ?

From the sequencing request, they told me that the coverage should be 30x.

Can you just guide me / get me started and I'll do the minion work afterwards.

Thanks.

rpetrovski commented 6 years ago

Here by pileup I mean an unusually high coverage of reads. When doing wgs pileups typically indicate an alignment artifact produced due to certain sequences that are abundant in the sample being insufficiently represented in the reference. When that happens, the aligner often places all reads that have such sequences into a few places in the reference, resulting in abnormally high local coverage spikes. I've seen pileups of several thousand X on a 30x samples.

By the looks of it, 22nd contig in your genome is chr9

[rpetrovski@ukch-tst-lnts12 bfiset]$ zcat Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz |grep '>'

1 dna:chromosome chromosome:GRCh38:1:1:248956422:1 REF 10 dna:chromosome chromosome:GRCh38:10:1:133797422:1 REF 11 dna:chromosome chromosome:GRCh38:11:1:135086622:1 REF 12 dna:chromosome chromosome:GRCh38:12:1:133275309:1 REF 13 dna:chromosome chromosome:GRCh38:13:1:114364328:1 REF 14 dna:chromosome chromosome:GRCh38:14:1:107043718:1 REF 15 dna:chromosome chromosome:GRCh38:15:1:101991189:1 REF 16 dna:chromosome chromosome:GRCh38:16:1:90338345:1 REF 17 dna:chromosome chromosome:GRCh38:17:1:83257441:1 REF 18 dna:chromosome chromosome:GRCh38:18:1:80373285:1 REF 19 dna:chromosome chromosome:GRCh38:19:1:58617616:1 REF 2 dna:chromosome chromosome:GRCh38:2:1:242193529:1 REF 20 dna:chromosome chromosome:GRCh38:20:1:64444167:1 REF 21 dna:chromosome chromosome:GRCh38:21:1:46709983:1 REF 22 dna:chromosome chromosome:GRCh38:22:1:50818468:1 REF 3 dna:chromosome chromosome:GRCh38:3:1:198295559:1 REF 4 dna:chromosome chromosome:GRCh38:4:1:190214555:1 REF 5 dna:chromosome chromosome:GRCh38:5:1:181538259:1 REF 6 dna:chromosome chromosome:GRCh38:6:1:170805979:1 REF 7 dna:chromosome chromosome:GRCh38:7:1:159345973:1 REF 8 dna:chromosome chromosome:GRCh38:8:1:145138636:1 REF 9 dna:chromosome chromosome:GRCh38:9:1:138394717:1 REF MT dna:chromosome chromosome:GRCh38:MT:1:16569:1 REF X dna:chromosome chromosome:GRCh38:X:1:156040895:1 REF Y dna:chromosome chromosome:GRCh38:Y:2781480:56887902:1 REF KI270728.1 dna:scaffold scaffold:GRCh38:KI270728.1:1:1872759:1 REF ...

Since we had crashes in the bin of length 20000 that starts at position 0 (ReferencePosition(22:0:f)bs 20000bl), it should be relatively easy to zoom into that region of chromosome 9 in a viewer such as IGV and check if there is a piling up of reads.

Please let me know how it goes.

One more thing. If possible, please try to keep the input fastq so that we can test the fix in the future.

Roman.

BenoitFiset commented 6 years ago

Hi Roman,

thanks for the help.

Should I see the pileups using the bam files that were generated from Isaac-04.18.01.18 or I use the partly generated bam file from the crashed / terminated from the run with Isaac-04.18.05.23 or should I align with other aligner like bwa and use this BAM file ?

For sure I'll keep the reference input fasta.

B.

rpetrovski commented 6 years ago

You need to use the successful one. Other aligners may or may not produce pileups in the same spots. Most likely not.

R.

BenoitFiset commented 6 years ago

Hi Roman,

FYI

Even with a 512G of memory server, I still get the error with version Isaac-04.18.05.23.. 

2018-08-20 15:29:19     [2aca52ffe700]  ERROR: Thread: 83 also caught an exception: 2018-Aug-20 15:29:19: Success: /home/bfiset/lw-project/Isaac-src/Isaac4-master/src/c++/lib/build/Build.cpp(634): Throw in funct
ion bool isaac::build::Build::handleBinAllocationFailure(bool, const isaac::alignment::BinMetadata&, const ExceptionType&, const ExceptionDataT&) [with ExceptionType = std::bad_alloc; ExceptionDataT = long unsig
ned int]
Dynamic exception type: boost::exception_detail::clone_impl<isaac::common::ThreadingException>
std::exception::what: ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.66563.0/Isaac_Temp_DIR/bin-00000022-130263695.dat
) blocking everything with std::bad_alloc : std::bad_alloc Error data: 1183078250 
: ERROR: Failing due to: BinMetadata(287510id ReferencePosition(22:0:f)bs 20000bl 894915818ds 0do 0se 1352658rs 1314084f /localscratch/bfiset.66563.0/Isaac_Temp_DIR/bin-00000022-130263695.dat) blocking everythin
g with std::bad_alloc : std::bad_alloc Error data: 1183078250

So Aligning with Isaac-04.18.01.18 which works

BenoitFiset commented 6 years ago

Hi Roman,

The files are humongous so haven't figured a way to open in IGV to look at the pileup.

rpetrovski commented 6 years ago

I think I have a solution. Let me do some validation, then I'll push a version you should be able to try.

Roman.

rpetrovski commented 6 years ago

@BenoitFiset, can you please try SAAC01346_branch. It should use less memory during bam generation than master.

Roman.

BenoitFiset commented 6 years ago

Hi Roman,

I'll try ASAP. I'll keep you posted.

Thanks.

BenoitFiset commented 6 years ago

Hi Roman,

I'm back on this... I still try SAAC01346_branch or Isaac-04.18.08.29 ?

Thanks,

B

rpetrovski commented 6 years ago

The branch please.

BenoitFiset commented 6 years ago

Ok Sir, will do.

Should I still use insane amount of memory to run ? Options are 32, 256, 512 Gb of ram

Thanks

rpetrovski commented 6 years ago

Can you please try the last that failed and then go down if you succeed.

BenoitFiset commented 6 years ago

What about the command line ?

This OK

/home/bfiset/lw-project/Isaac4-bin/bin/isaac-align 
--reference-genome /localscratch/bfiset.45188.0/IsaacIndex_GRCh38_V93/sorted-reference.xml 
--base-calls /home/bfiset/scratch/WGS/HI.4784-008_4763-006 
--memory-limit 250 --base-calls-format fastq --jobs 48 
--realign-gaps all 
--output-directory /localscratch/bfiset.45188.0/HI.4784-008_4763-006_Aligned_01 
--temp-directory /localscratch/bfiset.45188.0/Isaac_Temp_DIR

Do I now ignore the argument: --split-alignments with this branch ?

BenoitFiset commented 6 years ago

Hi Roman,

Good news, SAAC01346_branch is working with the files that were crashing Isaac. I was using the command line in the previous post. (and not using the argument --split-alignments).

Was running on usual 256Gb ram, 48 CPU server.

Any more tests ?

rpetrovski commented 6 years ago

Cheers for that. Sorry for not responding to your command line question. For some reason I did not get notification for that one.

If you could go down on memory-limit and see the lowest for which it works that would be awesome.

How long did your run take?

Roman.

BenoitFiset commented 6 years ago

Hi,

Il try on a 32GB of ram 24 CPU server.

Stats on the runs that work for WGS:

Start: 2018-11-06 00:51:52     [2b7e09b6f6c0]  Version: Isaac-SAAC01346.18.08.30
End:   2018-11-06 05:47:16     [2b7e09b6f6c0]  Saving workflow state done

Total bytes written: 116220200960 (109GiB, 52MiB/s)

B

BenoitFiset commented 6 years ago

Hi Roman,

It crashed and burned to the ground with 32GB of Ram....

Here are the begin of the job:

2018-11-06 10:32:56     [2b03bde186c0]  Version: Isaac-SAAC01346.18.08.30
[...SNIP Couple of Lines...]
018-11-06 10:32:56  [2b03bde186c0]  reads parsed: 2
2018-11-06 10:32:56     [2b03bde186c0]  Discovered data read: ReadMetadata(1, 150 [1, 150], 0id, 0off,1frc)
2018-11-06 10:32:56     [2b03bde186c0]  Discovered data read: ReadMetadata(2, 150 [152, 301], 1id, 150off,152frc)
2018-11-06 10:32:56     [2b03bde186c0]  Generated 'none' barcode: BarcodeMetadata(HMJGJCCXY,1,default,none,(0), 4294967295)
2018-11-06 10:32:56     [2b03bde186c0]  align: NUMA-aware memory management disabled.
2018-11-06 10:32:56     [2b03bde186c0]  align: Setting memory limit to 33285996544 bytes.
2018-11-06 10:32:56     [2b03bde186c0]  estimateOptimumFragmentsPerBin estimatedFragmentSize: 338
2018-11-06 10:32:56     [2b03bde186c0]  estimateOptimumFragmentsPerBin maxFragmentDedupedIndexBytes: 56
2018-11-06 10:32:56     [2b03bde186c0]  estimateOptimumFragmentsPerBin maxFragmentCompressedBytes: 338
2018-11-06 10:32:56     [2b03bde186c0]  estimateOptimumFragmentsPerBin availableMemory: 33285996544
2018-11-06 10:32:56     [2b03bde186c0]  estimateOptimumFragmentsPerBin fragmentMemoryRequirements: 732
2018-11-06 10:32:56     [2b03bde186c0]  estimateOptimumFragmentsPerBin minOverlap: 24
2018-11-06 10:32:56     [2b03bde186c0]  estimateOptimumFragmentsPerBin availableMemory / fragmentMemoryRequirements / minOverlap: 1894694
2018-11-06 10:32:56     [2b03bde186c0]  STAT: loadContigs  240230400vm 2989res
2018-11-06 10:33:02     [2b03bde186c0]  Generated 194 contigs of which 0 are decoys

Last entries in the black box flight recorder:

2018-11-06 10:33:13     [2b03bde186c0]  STAT: TemplateBuilder before shadowList_.reserve 4303740928vm 782674res
2018-11-06 10:33:13     [2b03bde186c0]  STAT: TemplateBuilder before bestCombinationPairInfo_.reserve 4304740352vm 782674res
2018-11-06 10:33:13     [2b03bde186c0]  STAT: TemplateBuilder before bestRescuedPair_.reserve 4304740352vm 782674res
2018-11-06 10:33:13     [2b03bde186c0]  STAT: TemplateBuilder before candidates_.reserve 4304740352vm 782674res
2018-11-06 10:33:13     [2b03bde186c0]  STAT: TemplateBuilder after candidates_.reserve 4304986112vm 782674res
2018-11-06 10:33:13     [2b03bde186c0]  STAT: Constructed match selector 4304986112vm 782674res
2018-11-06 10:33:32     [2b03bde186c0]  STAT: Constructing ReferenceHasher: for 16-mers  21560406016vm 4978323res
2018-11-06 10:35:04     [2b03bde186c0]   a:3308323 b:7048005 buckets:4294967296 and 2945831140 genome 16-mers  unique k-mers found 953082018 unique keys. maxUniqueKeys:953082018
std::bad_alloc

I'm guessing 32GB of RAM is not enough for a WGS alignement.... or is it ?

B.

rpetrovski commented 6 years ago

32 is on the low end for human WGS. One thing to make sure is that you see Genome offset type: j at the start of the log file. It indicates that support for long genomes is off and is the default when configuring isaac build unless genome-offset-max is set to larger than 4 gigabases.

If that all good, I guess you'd need to go up in -m. I think Illumina Basespace instance spec is in the order of 60G RAM.

BenoitFiset commented 6 years ago

Yep type is j:

2018-11-06 10:32:56     [2b03bde186c0]  Forcing LC_ALL to C
2018-11-06 10:32:56     [2b03bde186c0]  Version: Isaac-SAAC01346.18.08.30
2018-11-06 10:32:56     [2b03bde186c0]  Genome offset type: j
2018-11-06 10:32:56     [2b03bde186c0]  argc: 17 argv:

On this server farm all I have avail is 32GB, 256GB and 51GB.... So out of options to test memory between 32GB and 256GB.... Sorry...

rpetrovski commented 6 years ago

You limit the memory with --memory-limit. Isaac internally uses linux setrlimit, so it is guaranteed to not go over what you specity. You can run on 256 box with -m 50.

R.

BenoitFiset commented 6 years ago

Hi Roman,

--memory-limit 50 -----> Epic Crash and Burn !!!!

2018-11-07 11:29:52     [2b4493345740]  align: Setting memory limit to 53687091200 bytes.
2018-11-07 11:33:35     [2b4f182e7700]  Loaded  8000000 clusters of length 347
2018-11-07 11:33:35     [2b4f182e7700]  Fastq load thread terminated
std::bad_alloc

--memory-limit 60 -----> The bird crossed the ocean !!!!
Flight Took: 05:55:44

2018-11-07 11:41:42     [2b64beb4f740]  align: Setting memory limit to 64424509440 bytes.
2018-11-07 16:34:54     [2b64beb4f740]  Generating Build statistics
2018-11-07 16:34:55     [2b64beb4f740]  Generating Build statistics done
2018-11-07 16:34:55     [2b64beb4f740]  Generating the BAM files done

When used 256GB of Ram it took 05:55:11.... so consistant....

From the server stats, wondering on the ressource usage.... Doesn't seem to be using all the avail resources.... This just might not mean anything... but wanted to make you aware.

With 256GB ram:

Cores per node: 48
CPU Utilized: 4-04:00:49
CPU Efficiency: 35.20% of 11-20:08:48 core-walltime
Job Wall-clock time: 05:55:11
Memory Utilized: 42.09 GB
Memory Efficiency: 16.84% of 250.00 GB

With 60GB of ram:

Cores per node: 48
CPU Utilized: 4-03:01:44
CPU Efficiency: 34.80% of 11-20:35:12 core-walltime
Job Wall-clock time: 05:55:44
Memory Utilized: 44.70 GB
Memory Efficiency: 17.88% of 250.00 GB

Thanks for the help... Any next steps to help you ?

B.