ERROR: no align files in alignFolder

[olechnwin]

Hi, I decided to create a new post with the same title since I thought it would be easier. Please let me know if you want me to merge this thread to the other one.

So, I ran the runBNG denovo and got the no align files error. I'm copying the command I ran and the outputs. I've also copied part of "exp_pipelineReport_05.txt" below. Please let me know if you need to see anything else. Thank you again for your help.

export TMPDIR=/tmp
export SINGULARITY_LOCALCACHEDIR=/tmp
export SINGULARITY_CACHEDIR=/tmp
export SINGULARITY_TMPDIR=/tmp
SLURM_CPUS_TASK=20

cd ~/opt/runBNG
~/opt/runBNG/runBNG denovo \
        -t      ~/opt/bionano/tools/pipeline/Solve3.6.1_11162020/RefAligner/1.0 \
        -s      ~/opt/bionano/tools/pipeline/Solve3.6.1_11162020 \
        -b      ~/Data/A673_bionano/EwingSarcoma_November2019/EwingsSarcoma_Solve3.4_pipeline_results/output/all.bnx \
        -M      yes \
        -a      15 \
        -T      $((SLURM_CPUS_PER_TASK-2)) \
        -j      2 \
        -z      3137 \
        -r      ~/Data/A673_bionano/runBNG_rslt/hg19_DLE1_20kb_5labels.cmap \
        -o      ~/Data/A673_bionano/runBNG_rslt

========================================== De novo assembly starts =============================================
## Start date: Wed Apr  7 16:31:21 EDT 2021
## The data generation platform is: saphyr
## The bnx file is: /Data/A673_bionano/EwingSarcoma_November2019/EwingsSarcoma_Solve3.4_pipeline_results/output/all.bnx
## The digested reference is: /Data/A673_bionano/runBNG_rslt/hg19_DLE1_20kb_5labels.cmap
## The minimum molecule length is (Kb): 0.12
## The minimum label on a molecule is: 8
## Maximum backbone intensity is: 0.6
## The path to Bionano Solve folder is: /opt/bionano/tools/pipeline/Solve3.6.1_11162020
## The path to Bionano RefAligner folder is: /opt/bionano/tools/pipeline/Solve3.6.1_11162020/RefAligner/1.0
## The number of threads is: 18
## Large jobs maximum memory (GB) is: 128
## Small jobs maximum memory (GB) is: 8
## The number of threads for each subjob is: 2
## The number of iterations is: 5
## False Positive Density (/100Kb) [FP]: 2.0
## False Negative Rate (%/100) [FN]: 0.10
## ScalingSD (Kb^1/2) [sd]: 0.0
## SiteSD (Kb) [sf]: 
## RelativeSD [sr]: 0.03
## The genome size (Mb) is: 3137
## The xml file is: /opt/bionano/tools/pipeline/Solve3.6.1_11162020/RefAligner/1.0/optArguments_nonhaplotype_DLE1_saphyr_human.xml
## The output directory is: /Data/A673_bionano/runBNG_rslt

  Prerun Tests:
    0 ERRORS
    0 WARNINGS

Hostname: r1pl-hpcf-n15

Tools Version: N/A

Solve Version: N/A

Pipeline Version: 11646

RefAligner Version: 11643

Assembly ID: 91

  Pipeline start time: Wed Apr  7 16:32:22 2021

('checkScanScaling: autoNoise=', False)
Not performing autoNoise (see -y): setting doScanScale = False, hence not using _rescaled.bnx for BNX splits

Executing stage number 1 : AutoNoise + SplitBNX

Molecule Stats (/Data/A673_bionano/runBNG_rslt/all.bnx):
Total number of molecules: 7049432
Total length (Mbp)       : 772615.679
Average length (kbp)     :    109.600
Molecule N50 (kbp)       :    207.750
Label density (/100kb)   :      9.918

Sorting /Data/A673_bionano/runBNG_rslt/all.bnx into /Data/A673_bionano/runBNG_rslt/all_sorted

 Starting Multi-Threaded Process: SortBNX
  Running 1 jobs with 18 threads, sleepTime=0.01
   START    1:                        SortBNX,   2 Thr,    1 R,    1 T,    0 F,    0 Q
   STOP     1:                        SortBNX,   2 Thr,    0 R,    1 T,    1 F,    0 Q  TotalTime= 0h 1.38m  RunTime= 0h 1.38m  CPUload=133% host=NA
 Finished Multi-Threaded Process: SortBNX

calculateNPairwise: Sorted and filtered file has 2307193 molecules, 75318443.0 sites, 502407591866.0 bp length total. blocks= 14 -> 14 (N=2)

Increasing number of blocks from 2 to 14

Splitting BNX

Splitting bnx file: /Data/A673_bionano/runBNG_rslt/all_sorted.bnx

Removing 14 split bnx files

Removing:
/Data/A673_bionano/runBNG_rslt/all_1_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_2_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_3_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_4_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_5_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_6_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_7_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_8_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_9_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_10_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_11_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_12_of_14.bnx, //Data/A673_bionano/runBNG_rslt/all_13_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_14_of_14.bnx

 Starting Multi-Threaded Process: SplitBNX
  Running 1 jobs with 18 threads, sleepTime=0.01
   START    1:                       SplitBNX,   2 Thr,    1 R,    1 T,    0 F,    0 Q
   STOP     1:                       SplitBNX,   2 Thr,    0 R,    1 T,    1 F,    0 Q  TotalTime= 0h 1.28m  RunTime= 0h 1.29m  CPUload=128% host=NA
 Finished Multi-Threaded Process: SplitBNX

Expecting:
/Data/A673_bionano/runBNG_rslt/all_1_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_2_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_3_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_4_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_5_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_6_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_7_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_8_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_9_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_10_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_11_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_12_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_13_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_14_of_14.bnx

Executing stage number 2 : Pairwise + Alignmolvref

 Starting Multi-Threaded Process: Pairwise
  Running 105 jobs with 18 threads, sleepTime=0.01
   START    1:              Pairwise 1 of 105,  18 Thr,    1 R,  105 T,    0 F,  104 Q
jobName=pairwise1of105 : Invalid cpuString=
WARNING: Job Pairwise 1 of 105 has cpuTime= 0.0000, cpuPercent= 0.0000

   STOP     1:              Pairwise 1 of 105,  18 Thr,    0 R,  105 T,    1 F,  104 Q  TotalTime= 0h 0.00m  RunTime= 0h 0.00m  CPUload=0% host=NA
   START    2:              Pairwise 2 of 105,  18 Thr,    1 R,  105 T,    1 F,  103 Q
jobName=pairwise2of105 : Invalid cpuString=
WARNING: Job Pairwise 2 of 105 has cpuTime= 0.0000, cpuPercent= 0.0000

   STOP     2:              Pairwise 2 of 105,  18 Thr,    0 R,  105 T,    2 F,  103 Q  TotalTime= 0h 0.00m  RunTime= 0h 0.00m  CPUload=0% host=NA

#####There are other similar START/STOP#####
.
.
.
   START  105:            Pairwise 105 of 105,  18 Thr,    1 R,  105 T,  104 F,    0 Q
jobName=pairwise105of105 : Invalid cpuString=
WARNING: Job Pairwise 105 of 105 has cpuTime= 0.0000, cpuPercent= 0.0000

   STOP   105:            Pairwise 105 of 105,  18 Thr,    0 R,  105 T,  105 F,    0 Q  TotalTime= 0h 0.00m  RunTime= 0h 0.00m  CPUload=0% host=NA
 Finished Multi-Threaded Process: Pairwise

ERROR: no align files in alignFolder /Data/A673_bionano/runBNG_rslt/align

Warning/Error messages:
('warning', 'Found text in \\"/Data/A673_bionano/runBNG_rslt/all_sorted.stdout\\" after \\"END of output\\"\\n')
('warning', 'Found text in \\"/Data/A673_bionano/runBNG_rslt/all.stdout\\" after \\"END of output\\"\\n')
('warning', 'Could not open file \\"/Data/A673_bionano/runBNG_rslt/align/exppairwise1of105.stdout\\" tries=0 exception=[Errno 2] No such file or directory: \'/Data/A673_bionano/runBNG_rslt/align/exppairwise1of105.stdout\'\\n')
('error', 'job has not completed, see stdout=\\"/Data/A673_bionano/runBNG_rslt/align/exppairwise1of105.stdout\\"')
('warning', 'Could not open file \\"/Data/A673_bionano/runBNG_rslt/align/exppairwise2of105.stdout\\" tries=0 exception=[Errno 2] No such file or directory: \'/Data/A673_bionano/runBNG_rslt/align/exppairwise2of105.stdout\'\\n')

This is the content of "exp_pipelineReport_05.txt"

/opt/bionano/tools/pipeline/Solve3.6.1_11162020/Pipeline/1.0/pipelineCL.py -R -w -d -U -T 18 -j 2 -je 2 -jp 18 -J 2 -TJ 18 -Te 18 -Tp 18 -i 5 -t /opt/bionano/tools/pipeline/Solve3.6.1_11162020/RefAligner/1.0 -a /Data/A673_bionano/runBNG_rslt/optArguments_new.xml -l /Data/A673_bionano/runBNG_rslt -b /Data/A673_bionano/EwingSarcoma_November2019/EwingsSarcoma_Solve3.4_pipeline_results/output/all.bnx -r /Data/A673_bionano/runBNG_rslt/hg19_DLE1_20kb_5labels.cmap

Hostname: r1pl-hpcf-n15
Tools Version: N/A
Solve Version: N/A
Pipeline Version: 11646
RefAligner Version: 11643

Assembly ID: 91

  Pipeline start time: Wed Apr  7 16:32:22 2021

Not performing autoNoise (see -y): setting doScanScale = False, hence not using _rescaled.bnx for BNX splits
Executing stage number 1 : AutoNoise + SplitBNX

Sorting /Data/A673_bionano/runBNG_rslt/all.bnx into /Data/A673_bionano/runBNG_rslt/all_sorted
   SortBNX completed in:   83.0s  Result Found

  Completed 1 jobs on 18 threads
  Clock time:  0h 1.38m  CPU time  :  0h 1.38m
  Begin machine parsable segment for: SortBNX
time_header Job         Qtime       Usr Sys Elpsd   CPU MaxMem  EndTime
time_line   SortBNX     83.03481    102.47  7.99    1:23.03 133%    8810320 

calculateNPairwise: Sorted and filtered file has 2307193 molecules, 75318443.0 sites, 502407591866.0 bp length total. blocks= 14 -> 14 (N=2)

Increasing number of blocks from 2 to 14
Splitting BNX
Splitting bnx file: /Data/A673_bionano/runBNG_rslt/all_sorted.bnx

Removing 14 split bnx files
Removing:
/Data/A673_bionano/runBNG_rslt/all_1_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_2_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_3_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_4_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_5_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_6_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_7_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_8_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_9_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_10_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_11_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_12_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_13_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_14_of_14.bnx
   SplitBNX completed in:   76.9s  Result Found

  Completed 1 jobs on 18 threads
  Clock time:  0h 1.28m  CPU time  :  0h 1.28m
  Begin machine parsable segment for: SplitBNX
time_header Job         Qtime       Usr Sys Elpsd   CPU MaxMem  EndTime
time_line   SplitBNX    76.89917    94.14   4.78    1:16.89 128%    11519044    

Expecting:
/Data/A673_bionano/runBNG_rslt/all_1_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_2_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_3_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_4_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_5_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_6_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_7_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_8_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_9_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_10_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_11_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_12_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_13_of_14.bnx, /Data/A673_bionano/runBNG_rslt/all_14_of_14.bnx
Executing stage number 2 : Pairwise + Alignmolvref

Pairwise
/usr/bin/time -f %U %S  %E  %P  %M /opt/bionano/tools/pipeline/Solve3.6.1_11162020/RefAligner/1.0/RefAligner -i /Data/A673_bionano/runBNG_rslt/all_1_of_14.bnx -o /Data/A673_bionano/runBNG_rslt/align/exppairwise1of105 -usecolor 1 -FP 2.0 -FN 0.10 -sf  -sd 0.0 -sr 0.03 -res 3.1 -resSD 0.75 -maxmem 5.4 -TotalThreads -usecolor 1 -mres 0.9 -S 0.1 -A 8 -HashMaxHits 150 15 3000 -MaxSF 0.15 -MinFN 0.10 -MinFP 1.0 -MinSR 0.025 -MaxSE 0.5 -outlier 0.001 -outlierMax 40. -endoutlier 0 -maxEnd 90. -RepeatMask 5 0.01 -RepeatRec 0.7 0.6 1.4 -PVres 2 -alignscore -align_format 1 -maptype 0 -f -hashgen 5 4 2.2 1.2 0.05 3.0 1 1 -hash -insertThreads 4 -MinSD 0.0 -minlen 0.12 -minsites 8 -minlen 150 -maxsites 750 -maxSiteDensity 30 -maxContigSiteDensity 40 -MinFN 0.1 -MinFP 2.0 -MinSD 0.0 -MinSR 0.02 -MaxSF 0.15 -MaxSD 0.12 -MaxSR 0.03 -MinResSD 0.8 -T 6.37552e-13 -minlen 180.0 -stdout -stderr -maxthreads 18 -XmapStatRead /Data/A673_bionano/runBNG_rslt/molecule_stats.txt

   Pairwise 1 of 105 completed in:    0.0s  Result NOT Found: /Data/A673_bionano/runBNG_rslt/align/exppairwise1of105.align   Stdout INCOMPLETE: /Data/A673_bionano/runBNG_rslt/align/exppairwise1of105.stdout 
   OFFENDING ARGUMENTS:
/opt/bionano/tools/pipeline/Solve3.6.1_11162020/RefAligner/1.0/RefAligner -i /Data/A673_bionano/runBNG_rslt/all_1_of_14.bnx -o /Data/A673_bionano/runBNG_rslt/align/exppairwise1of105 -usecolor 1 -FP 2.0 -FN 0.10 -sf  -sd 0.0 -sr 0.03 -res 3.1 -resSD 0.75 -maxmem 5.4 -TotalThreads -usecolor 1 -mres 0.9 -S 0.1 -A 8 -HashMaxHits 150 15 3000 -MaxSF 0.15 -MinFN 0.10 -MinFP 1.0 -MinSR 0.025 -MaxSE 0.5 -outlier 0.001 -outlierMax 40. -endoutlier 0 -maxEnd 90. -RepeatMask 5 0.01 -RepeatRec 0.7 0.6 1.4 -PVres 2 -alignscore -align_format 1 -maptype 0 -f -hashgen 5 4 2.2 1.2 0.05 3.0 1 1 -hash -insertThreads 4 -MinSD 0.0 -minlen 0.12 -minsites 8 -minlen 150 -maxsites 750 -maxSiteDensity 30 -maxContigSiteDensity 40 -MinFN 0.1 -MinFP 2.0 -MinSD 0.0 -MinSR 0.02 -MaxSF 0.15 -MaxSD 0.12 -MaxSR 0.03 -MinResSD 0.8 -T 6.37552e-13 -minlen 180.0 -stdout -stderr -maxthreads 18 -XmapStatRead /Data/A673_bionano/runBNG_rslt/molecule_stats.txt
   Pairwise 2 of 105 completed in:    0.0s  Result NOT Found: /Data/A673_bionano/runBNG_rslt/align/exppairwise2of105.align   Stdout INCOMPLETE: /Data/A673_bionano/runBNG_rslt/align/exppairwise2of105.stdout 

[olechnwin]

just a follow-up. In case this provides more info. I tried changing the label density from 15 to 9 and minimum labels from 8 to 3 but still got the same error.

[yyx8671]

Thanks @olechnwin for posting this. I found a bug in the script.

1352     if [[ -z "$sf" ]]; then
1353         if [[ "$plt" == "irys" ]]; then
1354             sf=0.15
1355         else
1356             len=0.12
1357         fi
1358     fi

len=0.12 should be sf=0.12. The bug has been fixed in the repo and please update your clone and have a try

[olechnwin]

@yyx8671,

Thank you for that quick fix. Looks like it's currently running fine.

[olechnwin]

@yyx8671 ,

My runBNG denovo got an out of memory error. I'm trying to figure out how much memory I should request. I'm hoping you can provide some insight. This is the setting I have:

## The number of threads is: 18
## Large jobs maximum memory (GB) is: 128
## Small jobs maximum memory (GB) is: 8
## The number of threads for each subjob is: 2

From our job statistics output,

refineB1: waited 27000.00 seconds for job completion: UnsubmittedJobs= 514, ActiveJobs= 9, FinishedJobs= 8
slurmstepd: error: Detected 1 oom-kill event(s) in step 355938.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
Job Statistics for 355938:
           JobID       User               Start                 End    Elapsed     MaxRSS   TotalCPU      State Exit        NodeList                                  ReqTRES 
---------------- ---------- ------------------- ------------------- ---------- ---------- ---------- ---------- ---- --------------- ---------------------------------------- 
          355938     xxt050 2021-04-09T08:28:09 2021-04-10T20:28:35 1-12:00:26            27-04:01:+    TIMEOUT  0:0   r1pl-hpcf-n17     billing=20,cpu=20,mem=156240M,node=1 
    355938.batch            2021-04-09T08:28:09 2021-04-10T20:29:05 1-12:00:56 142602.17M 27-04:01:+ OUT_OF_ME+ 0:1+   r1pl-hpcf-n17                                          
   355938.extern            2021-04-09T08:28:09 2021-04-10T20:28:35 1-12:00:26          0  00:00.001  COMPLETED  0:0   r1pl-hpcf-n17                                          
CPU Efficiency: 90.54% of 30-00:08:40 core-walltime

Can you please explain with the large jobs and small jobs how much memory it needs with the threads settings ? For example, with thread=18 and max memory large jobs = 128GB, max memory small jobs = 8GB and thread subjob=2, does that mean it requires a total of 18 128GB + 2 8GB memory? Thanks!

[yyx8671]

Hi @olechnwin，

This depends on your data, for instance the coverage that your data have. You may ask for '196 Gb' RAM in your slurm script to run if your server can provide.

[olechnwin]

Hi @yyx8671,

Thanks so much for your quick reply. Can I ask where does 196Gb come from? The reason I'm asking is that my server has a total of 32 cpus and ~250Gb memory. I was requesting for 20 cpus which by default gets me ~156Gb memory.

[yyx8671]

Hi @olechnwin,

For the default setting in the Bionano configration file, the max RAM required is '248 Gb'. Here I suggested a rough number (196 Gb) for your consideration, as from my previous tests, '196 Gb' was sufficient for my tasks.

Most of the steps use a small memory and only some steps use a huge memory. If you cannot request a large memory on your server, you may reduce the CPUs and the total number of parallel subjobs, but this will consume more walltime to complete your task.

[olechnwin]

Hi @yyx8671 ,

I see. Thanks so much for your explanation. That helps.