search

brwnj / smoove-nf

Nextflow implementation of the smoove workflow and other tools for SV calling and QC
MIT License
12 stars 2 forks source link

interesting pipeline but singularity error #4

Open colindaven opened 4 years ago

colindaven commented 4 years ago

Hi, I'm wondering if I don't have the singularity scratch dir configured or if I have an older singularity version?

ls: cannot access '/var/lib/singularity/mnt/final/scratch': No such file or directory

singularity --version
2.6.1-dist

thanks Colin


nextflow run brwnj/smoove-nf -latest -profile singularity --fasta /lager2/rcug/seqres/RN/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa --gff /lager2/rcug/seqres/RN/annotation/Rnor6/Rattus_norvegicus.Rnor_6.0.87.gff3  --bams /lager2/rcug/2017/922E/*.calmd.bam
Picked up _JAVA_OPTIONS: -Dhttp.proxyHost=172.24.2.50 -Dhttp.proxyPort=8080 -Dhttps.proxyHost=172.24.2.50 -Dhttps.proxyPort=8080
N E X T F L O W  ~  version 19.10.0
Pulling brwnj/smoove-nf ...
Already-up-to-date
Launching `brwnj/smoove-nf` [reverent_torvalds] - revision: 4e30cb6bbc [master]

Project            (--project)       : sites
WARN: Access to undefined parameter `bed` -- Initialise it to a default value eg. `params.bed = some_value`
Excluded chroms    (--exclude)       : ~^HLA,~^hs,~:,~^GL,~M,~EBV,~^NC,~^phix,~decoy,~random$,~Un,~hap,~_alt$
Reference fasta    (--fasta)         : /lager2/rcug/seqres/RN/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa
Sex chromosomes    (--sexchroms)     : X,Y
Alignments         (--bams)          : /lager2/rcug/2017/922E/1AR1_bwa_s.bam.calmd.bam
Indexes                              : /lager2/rcug/2017/922E/1AR1_bwa_s.bam.calmd.bam.bai
Annotation GFF     (--gff)           : /lager2/rcug/seqres/RN/annotation/Rnor6/Rattus_norvegicus.Rnor_6.0.87.gff3
Sensitive          (--sensitive)     : false
Output             (--outdir)        : ./results

executor >  local (4)
[34/57bedb] process > smoove_call (1)     [100%] 2 of 2, failed: 2
[-        ] process > smoove_merge        -
[-        ] process > smoove_genotype     -
[-        ] process > smoove_square       -
[bf/72f1e8] process > run_indexcov        [100%] 2 of 2, failed: 2 ✘
[-        ] process > merge_peds          -
[-        ] process > build_covviz_report -
[-        ] process > somalier_extract    -
[-        ] process > somalier_relate     -
[-        ] process > build_report        -
[a5/3ac7a4] NOTE: Process `smoove_call (1)` terminated with an error exit status (255) -- Execution is retried (1)
[1d/63a330] NOTE: Process `run_indexcov` terminated with an error exit status (255) -- Execution is retried (1)
WARN: Killing pending tasks (1)
Error executing process > 'run_indexcov'

Caused by:
  Process `run_indexcov` terminated with an error exit status (255)

Command executed:

  goleft indexcov --sex X,Y --excludepatt "~^HLA,~^hs,~:,~^GL,~M,~EBV,~^NC,~^phix,~decoy,~random$,~Un,~hap,~_alt$" --directory sites --fai Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.fai 1AR1_bwa_s.bam.calmd.bam.bai
  mv sites/* .

Command exit status:
  255

Command output:
  (empty)

Command error:
  ERROR  : Failed to get real path of /var/lib/singularity/mnt/final/scratch: No such file or directory
  ABORT  : Retval = 255

Work dir:
  /ngsssd1/rcug/922E_SVs_2019/work/bf/72f1e8cee1e1327c30453d4cd61df8

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

(wochenende) rcug@hpc-rc02:/ngsssd1/rcug/922E_SVs_2019$ ls /var/lib/singularity/mnt/final/scratch
ls: cannot access '/var/lib/singularity/mnt/final/scratch': No such file or directory
brwnj commented 4 years ago

It's true that your singularity version is very old. Your system admins should absolutely update. The current version is 3.5.3.

Why not configure singularity to use a scratch directory that does exist?

You may also want to configure your local nextflow profile for your system in order to change the bind parameter, e.g.:

profiles {
    singularity {
        singularity.runOptions = '--bind /lager2 --bind /ngsssd1'
        singularity.enabled = true
    }
}

See: https://www.nextflow.io/docs/latest/config.html#configuration-file.

colindaven commented 4 years ago

Hi, thanks for that. I upgraded singularity to 3.5.3.

I could change the relevant mounts (bind) by downloading your example nextflow.config to the current working directory and like you say adding those binds.

Here an example:


profiles {
    docker {
        docker.enabled = true
    }
    singularity {
        // bind ngsssd1 as scratch in container
        singularity.runOptions = '--bind /ngsssd1/rcug/tmp:/scratch,/working2/rcug/scratch/tmp,/lager2/rcug'
        singularity.enabled = true
    }
    none {}
}

proxy problem: I also solved the failure to pull other containers from within singularity by pulling externally on the shell with singularity pull docker://brwnj/covviz:v1.1.5

colindaven commented 4 years ago

Sadly its now complaining about FileNotFoundError: [Errno 2] No such file or directory: 'sites-indexcov.ped'

I don't see this on the required input files and don't know exactly what it wants/needs.

cheers

brwnj commented 4 years ago

Can you attach your .nextflow.log for the previous message?

colindaven commented 4 years ago

Here you go, thanks. Is it something to do with the sex chromosomes ? - I guess it shouldn't be, the rat genome looks like this:

less *.fai
1       282763074       58      60      61
10      112626471       287475910       60      61
11      90463843        401979548       60      61
12      52716770        493951181       60      61
13      114033958       547546624       60      61
14      115493446       663481208       60      61
15      111246239       780899605       60      61
16      90668790        894000007       60      61
17      90843779        986180003       60      61
18      88201929        1078537904      60      61
19      62275575        1168209925      60      61
2       266435125       1231523485      60      61
20      56205956        1502399255      60      61
3       177699992       1559542035      60      61
4       184226339       1740203752      60      61
5       173707219       1927500588      60      61
6       147991367       2104102986      60      61
7       145729302       2254560934      60      61
8       133307652       2402719116      60      61
9       122095297       2538248621      60      61
MT      16313   2662378896      60      61
X       159970021       2662395539      60      61
Y       3310458 2825031784      60      61
KL568162.1      10937627        2828397488      60      61
KL568139.1      9752924 2839517479      60      61
KL568161.1      7627431 2849433022      60      61
KL568148.1      6483517 2857187647      60      61
KL568157.1      5447879 2863779293      60      61
KL568160.1      4317250 2869318040      60      61
KL568151.1      3931032 2873707315      60      61
KL568149.1      3292674 2877703935      60      61
KL568141.1      2733323 2881051557      60      61
KL568156.1      2193381 2883830506      60      61
KL568159.1      1977169 2886060514      60      61
KL568155.1      1365599 2888070706      60      61
KL568140.1      1351663 2889459135      60      61

Mar-31 16:12:59.511 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: build_covviz_report; status: COMPLETED; exit: 1; error: -; workDir: /ngsssd1/rcug/singularity/smoove-nf/work/60/7bfda7a90e493e11b5c742a453ddf4]
Mar-31 16:12:59.522 [Task monitor] INFO  nextflow.processor.TaskProcessor - [60/7bfda7] NOTE: Process `build_covviz_report` terminated with an error exit status (1) -- Execution is retried (1)
Mar-31 16:12:59.543 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-31 16:12:59.544 [Task submitter] INFO  nextflow.Session - [9f/7adc81] Re-submitted process > build_covviz_report
Mar-31 16:13:01.161 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: build_covviz_report; status: COMPLETED; exit: 1; error: -; workDir: /ngsssd1/rcug/singularity/smoove-nf/work/9f/7adc81fed2d60bcf79d2dcd96b4dd1]
Mar-31 16:13:01.183 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'build_covviz_report'

Caused by:
  Process `build_covviz_report` terminated with an error exit status (1)

Command executed:

  covviz --min-samples 8 --sex-chroms X,Y --exclude '~^HLA,~^hs,~:,~^GL,~M,~EBV,~^NC,~^phix,~decoy,~random$,~Un,~hap,~_alt$'         --skip-norm --z-threshold 3.5 --distance-threshold 150000         --slop 500000 --ped sites-indexcov.ped --gff Rattus_norvegicus.Rnor_6.0.87.gff3 sites-indexcov.bed.gz

Command exit status:
  1

Command output:
  (empty)

Command error:
  [INFO] parsing bed file (sites-indexcov.bed.gz)
  Traceback (most recent call last):
    File "/opt/conda/envs/env/bin/covviz", line 10, in <module>
      sys.exit(cli())
    File "/opt/conda/envs/env/lib/python3.7/site-packages/covviz/covviz.py", line 222, in cli
      args.skip_norm,
    File "/opt/conda/envs/env/lib/python3.7/site-packages/covviz/bed.py", line 273, in parse_bed
      groups = parse_sex_groups(ped, sample_col, sex_col)
    File "/opt/conda/envs/env/lib/python3.7/site-packages/covviz/bed.py", line 143, in parse_sex_groups
      with open(filename) as fh:
  FileNotFoundError: [Errno 2] No such file or directory: 'sites-indexcov.ped'

Work dir:
  /ngsssd1/rcug/singularity/smoove-nf/work/9f/7adc81fed2d60bcf79d2dcd96b4dd1

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
Mar-31 16:13:01.189 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `build_covviz_report` terminated with an error exit status (1)
Mar-31 16:13:01.218 [Task monitor] DEBUG nextflow.Session - The following nodes are still active:
[process] smoove_merge
  status=ACTIVE
  port 0: (value) OPEN  ; channel: vcf
  port 1: (value) OPEN  ; channel: idx
  port 2: (value) bound ; channel: fasta
  port 3: (value) bound ; channel: faidx
  port 4: (cntrl) -     ; channel: $

[process] smoove_genotype
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: SMOOVE_KEEP_ALL
  port 1: (queue) OPEN  ; channel: -
  port 2: (value) OPEN  ; channel: sites
  port 3: (value) bound ; channel: fasta
  port 4: (value) bound ; channel: faidx
  port 5: (cntrl) -     ; channel: $

[process] smoove_square
  status=ACTIVE
  port 0: (value) OPEN  ; channel: vcf
  port 1: (value) OPEN  ; channel: idx
  port 2: (value) bound ; channel: gff
  port 3: (cntrl) -     ; channel: $

[process] build_report
  status=ACTIVE
  port 0: (value) OPEN  ; channel: sequence_count
  port 1: (value) OPEN  ; channel: variant_count
  port 2: (value) OPEN  ; channel: vcf
  port 3: (value) bound ; channel: pedfile
  port 4: (value) OPEN  ; channel: variant_html
  port 5: (cntrl) -     ; channel: $

Mar-31 16:13:01.221 [main] DEBUG nextflow.Session - Session await > all process finished
Mar-31 16:13:01.221 [main] DEBUG nextflow.Session - Session await > all barriers passed
Mar-31 16:13:01.234 [main] WARN  n.processor.TaskPollingMonitor - Killing pending tasks (1)
Mar-31 16:13:01.257 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=2; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=1; abortedCount=1; succeedDuration=2.7s; failedDuration=3.5s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=2; peakCpus=2; peakMemory=16 GB; ]
Mar-31 16:13:01.257 [main] DEBUG nextflow.trace.TraceFileObserver - Flow completing -- flushing trace file
Mar-31 16:13:01.261 [main] DEBUG nextflow.trace.ReportObserver - Flow completing -- rendering html report
Mar-31 16:13:01.302 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
  [{"cpuUsage":{"mean":91.4,"min":91.4,"q1":91.4,"q2":91.4,"q3":91.4,"max":91.4,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"process":"run_indexcov","mem":{"mean":56840192,"min":56840192,"q1":56840192,"q2":56840192,"q3":56840192,"max":56840192,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"memUsage":{"mean":0.66,"min":0.66,"q1":0.66,"q2":0.66,"q3":0.66,"max":0.66,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"timeUsage":null,"vmem":{"mean":140840960,"min":140840960,"q1":140840960,"q2":140840960,"q3":140840960,"max":140840960,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"reads":{"mean":8431460,"min":8431460,"q1":8431460,"q2":8431460,"q3":8431460,"max":8431460,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"cpu":{"mean":91.4,"min":91.4,"q1":91.4,"q2":91.4,"q3":91.4,"max":91.4,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"time":{"mean":2731,"min":2731,"q1":2731,"q2":2731,"q3":2731,"max":2731,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"writes":{"mean":6472897,"min":6472897,"q1":6472897,"q2":6472897,"q3":6472897,"max":6472897,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"}},{"cpuUsage":null,"process":"build_covviz_report","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":{"mean":1752,"min":1550,"q1":1651,"q2":1752,"q3":1853,"max":1954,"minLabel":"build_covviz_report","maxLabel":"build_covviz_report","q1Label":"build_covviz_report","q2Label":"build_covviz_report","q3Label":"build_covviz_report"},"writes":null},{"cpuUsage":null,"process":"smoove_call","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":null,"writes":null}]
Mar-31 16:13:02.196 [main] DEBUG nextflow.trace.TimelineObserver - Flow completing -- rendering html timeline
Mar-31 16:13:02.414 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Mar-31 16:13:02.744 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
brwnj commented 4 years ago

May I see the invocation of the workflow? It'll be at the top of the file I asked for (.nextflow.log). The file sites-indexcov.ped is coming from run_indexcov task and should always be present, but I need more info to debug.

colindaven commented 4 years ago

Apologies..

cat .nextflow.log
Mar-31 16:12:47.509 [main] DEBUG nextflow.cli.Launcher - Setting HTTP proxy: [proxy.mh-hannover.de, 8080]
Mar-31 16:12:47.640 [main] DEBUG nextflow.cli.Launcher - Setting HTTPS proxy: [proxy.mh-hannover.de, 8080]
Mar-31 16:12:47.641 [main] DEBUG nextflow.cli.Launcher - Setting http proxy: [proxy.mh-hannover.de, 8080]
Mar-31 16:12:47.641 [main] DEBUG nextflow.cli.Launcher - Setting https proxy: [proxy.mh-hannover.de, 8080]
Mar-31 16:12:47.641 [main] DEBUG nextflow.cli.Launcher - $> nextflow run brwnj/smoove-nf -latest -profile singularity --fasta /lager2/rcug/seqres/RN/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa --gff /lager2/rcug/seqres/RN/annotation/Rnor6/Rattus_norvegicus.Rnor_6.0.87.gff3 --bams /lager2/rcug/2017/922E/DRAGEN_BAM/1AR1_RG.markdup.calmd.bam /lager2/rcug/2017/922E/DRAGEN_BAM/iddm_RG.markdup.calmd.bam
Mar-31 16:12:47.869 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 20.01.0
Mar-31 16:12:49.098 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/rcug/.nextflow/assets/brwnj/smoove-nf/.git/config; branch: master; remote: origin; url: https://github.com/brwnj/smoove-nf.git
Mar-31 16:12:49.115 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/rcug/.nextflow/assets/brwnj/smoove-nf/.git/config; branch: master; remote: origin; url: https://github.com/brwnj/smoove-nf.git
Mar-31 16:12:49.116 [main] INFO  nextflow.cli.CmdRun - Pulling brwnj/smoove-nf ...
Mar-31 16:12:49.116 [main] DEBUG nextflow.scm.AssetManager - Pull pipeline brwnj/smoove-nf  -- Using local path: /home/rcug/.nextflow/assets/brwnj/smoove-nf
Mar-31 16:12:51.240 [main] INFO  nextflow.cli.CmdRun -  Already-up-to-date
Mar-31 16:12:51.245 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/rcug/.nextflow/assets/brwnj/smoove-nf/.git/config; branch: master; remote: origin; url: https://github.com/brwnj/smoove-nf.git
Mar-31 16:12:51.245 [main] INFO  nextflow.cli.CmdRun - Launching `brwnj/smoove-nf` [confident_coulomb] - revision: 99dc059c62 [master]
Mar-31 16:12:51.299 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/rcug/.nextflow/assets/brwnj/smoove-nf/nextflow.config
Mar-31 16:12:51.300 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /ngsssd1/rcug/singularity/smoove-nf/nextflow.config
Mar-31 16:12:51.300 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/rcug/.nextflow/assets/brwnj/smoove-nf/nextflow.config
Mar-31 16:12:51.301 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /ngsssd1/rcug/singularity/smoove-nf/nextflow.config
Mar-31 16:12:51.312 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Mar-31 16:12:51.464 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `singularity`
Mar-31 16:12:51.572 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [singularity, none, docker]
Mar-31 16:12:51.717 [main] DEBUG nextflow.Session - Session uuid: 38a38d1b-521f-4483-bb8d-2fdac49de4b1
Mar-31 16:12:51.718 [main] DEBUG nextflow.Session - Run name: confident_coulomb
Mar-31 16:12:51.718 [main] DEBUG nextflow.Session - Executor pool size: 24
Mar-31 16:12:51.768 [main] DEBUG nextflow.cli.CmdRun -
  Version: 20.01.0 build 5264
  Created: 12-02-2020 10:14 UTC (11:14 CEST)
  System: Linux 4.15.0-70-generic
  Runtime: Groovy 2.5.8 on OpenJDK 64-Bit Server VM 1.8.0_121-b15
  Encoding: UTF-8 (UTF-8)
  Process: 12458@hpc01 [127.0.1.1]
  CPUs: 24 - Mem: 47.2 GB (459.1 MB) - Swap: 3.7 GB (2.2 GB)
Mar-31 16:12:51.864 [main] DEBUG nextflow.Session - Work-dir: /ngsssd1/rcug/singularity/smoove-nf/work [nfs]
Mar-31 16:12:51.864 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/rcug/.nextflow/assets/brwnj/smoove-nf/bin
Mar-31 16:12:51.911 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Mar-31 16:12:51.914 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Mar-31 16:12:52.290 [main] DEBUG nextflow.Session - Session start invoked
Mar-31 16:12:52.296 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /ngsssd1/rcug/singularity/smoove-nf/results/logs/trace.txt
Mar-31 16:12:52.946 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Mar-31 16:12:52.958 [main] DEBUG nextflow.Session - Workflow process names [dsl1]: somalier_relate, build_report, run_indexcov, smoove_call, build_covviz_report, smoove_genotype, smoove_square, merge_peds, somalier_extract, smoove_merge
Mar-31 16:12:53.011 [main] INFO  nextflow.Nextflow -

Mar-31 16:12:53.016 [main] INFO  nextflow.Nextflow - Project            (--project)       : sites
Mar-31 16:12:53.018 [main] WARN  nextflow.script.ScriptBinding - Access to undefined parameter `bed` -- Initialise it to a default value eg. `params.bed = some_value`
Mar-31 16:12:53.019 [main] INFO  nextflow.Nextflow - Excluded chroms    (--exclude)       : ~^HLA,~^hs,~:,~^GL,~M,~EBV,~^NC,~^phix,~decoy,~random$,~Un,~hap,~_alt$
Mar-31 16:12:53.019 [main] INFO  nextflow.Nextflow - Reference fasta    (--fasta)         : /lager2/rcug/seqres/RN/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa
Mar-31 16:12:53.019 [main] INFO  nextflow.Nextflow - Sex chromosomes    (--sexchroms)     : X,Y
Mar-31 16:12:53.020 [main] INFO  nextflow.Nextflow - Alignments         (--bams)          : /lager2/rcug/2017/922E/DRAGEN_BAM/1AR1_RG.markdup.calmd.bam
Mar-31 16:12:53.020 [main] INFO  nextflow.Nextflow - Indexes                              : /lager2/rcug/2017/922E/DRAGEN_BAM/1AR1_RG.markdup.calmd.bam.bai
Mar-31 16:12:53.020 [main] INFO  nextflow.Nextflow - Annotation GFF     (--gff)           : /lager2/rcug/seqres/RN/annotation/Rnor6/Rattus_norvegicus.Rnor_6.0.87.gff3
Mar-31 16:12:53.020 [main] INFO  nextflow.Nextflow - Sensitive          (--sensitive)     : false
Mar-31 16:12:53.021 [main] INFO  nextflow.Nextflow - Output             (--outdir)        : ./results
Mar-31 16:12:53.021 [main] INFO  nextflow.Nextflow -

Mar-31 16:12:53.258 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:smoove_call` matches process smoove_call
Mar-31 16:12:53.262 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.262 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.270 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Mar-31 16:12:53.277 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=24; memory=47.2 GB; capacity=24; pollInterval=100ms; dumpInterval=5m
Mar-31 16:12:53.352 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > smoove_call -- maxForks: 24; blocking: false
Mar-31 16:12:53.512 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:smoove_merge` matches process smoove_merge
Mar-31 16:12:53.514 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.514 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.524 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > smoove_merge -- maxForks: 24; blocking: false
Mar-31 16:12:53.531 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:smoove_genotype` matches process smoove_genotype
Mar-31 16:12:53.532 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.532 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.536 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > smoove_genotype -- maxForks: 24; blocking: false
Mar-31 16:12:53.550 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:smoove_square` matches process smoove_square
Mar-31 16:12:53.551 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.551 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.554 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > smoove_square -- maxForks: 24; blocking: false
Mar-31 16:12:53.579 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:run_indexcov` matches process run_indexcov
Mar-31 16:12:53.579 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.580 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.584 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > run_indexcov -- maxForks: 24; blocking: false
Mar-31 16:12:53.597 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:covviz` matches label `covviz` for process with name merge_peds
Mar-31 16:12:53.599 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.599 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.602 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > merge_peds -- maxForks: 24; blocking: false
Mar-31 16:12:53.621 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:covviz` matches label `covviz` for process with name build_covviz_report
Mar-31 16:12:53.622 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.622 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.624 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > build_covviz_report -- maxForks: 24; blocking: false
Mar-31 16:12:53.633 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:somalier` matches label `somalier` for process with name somalier_extract
Mar-31 16:12:53.634 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.634 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.637 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > somalier_extract -- maxForks: 24; blocking: false
Mar-31 16:12:53.638 [Actor Thread 6] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=docker://brentp/smoove:v0.2.5; path=/tmp/singularity/brentp-smoove-v0.2.5.img
Mar-31 16:12:53.649 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:somalier` matches label `somalier` for process with name somalier_relate
Mar-31 16:12:53.650 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.650 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.654 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > somalier_relate -- maxForks: 24; blocking: false
Mar-31 16:12:53.662 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-31 16:12:53.662 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-31 16:12:53.664 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > build_report -- maxForks: 24; blocking: false
Mar-31 16:12:53.668 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
Mar-31 16:12:53.668 [main] DEBUG nextflow.Session - Session await
Mar-31 16:12:53.781 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-31 16:12:53.788 [Task submitter] INFO  nextflow.Session - [cd/ace1ed] Submitted process > smoove_call (1)
Mar-31 16:12:53.813 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-31 16:12:53.814 [Task submitter] INFO  nextflow.Session - [8a/dd2dcf] Submitted process > run_indexcov
Mar-31 16:12:57.391 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: run_indexcov; status: COMPLETED; exit: 0; error: -; workDir: /ngsssd1/rcug/singularity/smoove-nf/work/8a/dd2dcf797e4c001a07bb807e9b520a]
Mar-31 16:12:57.464 [Actor Thread 5] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=docker://brwnj/covviz:v1.1.5; path=/tmp/singularity/brwnj-covviz-v1.1.5.img
Mar-31 16:12:57.475 [Task monitor] DEBUG n.util.BlockingThreadExecutorFactory - Thread pool name=FileTransfer; maxThreads=72; maxQueueSize=216; keepAlive=1m
Mar-31 16:12:57.533 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-31 16:12:57.533 [Task submitter] INFO  nextflow.Session - [60/7bfda7] Submitted process > build_covviz_report
Mar-31 16:12:59.511 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: build_covviz_report; status: COMPLETED; exit: 1; error: -; workDir: /ngsssd1/rcug/singularity/smoove-nf/work/60/7bfda7a90e493e11b5c742a453ddf4]
Mar-31 16:12:59.522 [Task monitor] INFO  nextflow.processor.TaskProcessor - [60/7bfda7] NOTE: Process `build_covviz_report` terminated with an error exit status (1) -- Execution is retried (1)
Mar-31 16:12:59.543 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-31 16:12:59.544 [Task submitter] INFO  nextflow.Session - [9f/7adc81] Re-submitted process > build_covviz_report
Mar-31 16:13:01.161 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: build_covviz_report; status: COMPLETED; exit: 1; error: -; workDir: /ngsssd1/rcug/singularity/smoove-nf/work/9f/7adc81fed2d60bcf79d2dcd96b4dd1]
Mar-31 16:13:01.183 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'build_covviz_report'

Caused by:
  Process `build_covviz_report` terminated with an error exit status (1)

Command executed:

  covviz --min-samples 8 --sex-chroms X,Y --exclude '~^HLA,~^hs,~:,~^GL,~M,~EBV,~^NC,~^phix,~decoy,~random$,~Un,~hap,~_alt$'         --skip-norm --z-threshold 3.5 --distance-threshold 150000         --slop 500000 --ped sites-indexcov.ped --gff Rattus_norvegicus.Rnor_6.0.87.gff3 sites-indexcov.bed.gz

Command exit status:
  1

Command output:
  (empty)

Command error:
  [INFO] parsing bed file (sites-indexcov.bed.gz)
  Traceback (most recent call last):
    File "/opt/conda/envs/env/bin/covviz", line 10, in <module>
      sys.exit(cli())
    File "/opt/conda/envs/env/lib/python3.7/site-packages/covviz/covviz.py", line 222, in cli
      args.skip_norm,
    File "/opt/conda/envs/env/lib/python3.7/site-packages/covviz/bed.py", line 273, in parse_bed
      groups = parse_sex_groups(ped, sample_col, sex_col)
    File "/opt/conda/envs/env/lib/python3.7/site-packages/covviz/bed.py", line 143, in parse_sex_groups
      with open(filename) as fh:
  FileNotFoundError: [Errno 2] No such file or directory: 'sites-indexcov.ped'

Work dir:
  /ngsssd1/rcug/singularity/smoove-nf/work/9f/7adc81fed2d60bcf79d2dcd96b4dd1

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
Mar-31 16:13:01.189 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `build_covviz_report` terminated with an error exit status (1)
Mar-31 16:13:01.218 [Task monitor] DEBUG nextflow.Session - The following nodes are still active:
[process] smoove_merge
  status=ACTIVE
  port 0: (value) OPEN  ; channel: vcf
  port 1: (value) OPEN  ; channel: idx
  port 2: (value) bound ; channel: fasta
  port 3: (value) bound ; channel: faidx
  port 4: (cntrl) -     ; channel: $

[process] smoove_genotype
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: SMOOVE_KEEP_ALL
  port 1: (queue) OPEN  ; channel: -
  port 2: (value) OPEN  ; channel: sites
  port 3: (value) bound ; channel: fasta
  port 4: (value) bound ; channel: faidx
  port 5: (cntrl) -     ; channel: $

[process] smoove_square
  status=ACTIVE
  port 0: (value) OPEN  ; channel: vcf
  port 1: (value) OPEN  ; channel: idx
  port 2: (value) bound ; channel: gff
  port 3: (cntrl) -     ; channel: $

[process] build_report
  status=ACTIVE
  port 0: (value) OPEN  ; channel: sequence_count
  port 1: (value) OPEN  ; channel: variant_count
  port 2: (value) OPEN  ; channel: vcf
  port 3: (value) bound ; channel: pedfile
  port 4: (value) OPEN  ; channel: variant_html
  port 5: (cntrl) -     ; channel: $

Mar-31 16:13:01.221 [main] DEBUG nextflow.Session - Session await > all process finished
Mar-31 16:13:01.221 [main] DEBUG nextflow.Session - Session await > all barriers passed
Mar-31 16:13:01.234 [main] WARN  n.processor.TaskPollingMonitor - Killing pending tasks (1)
Mar-31 16:13:01.257 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=2; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=1; abortedCount=1; succeedDuration=2.7s; failedDuration=3.5s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=2; peakCpus=2; peakMemory=16 GB; ]
Mar-31 16:13:01.257 [main] DEBUG nextflow.trace.TraceFileObserver - Flow completing -- flushing trace file
Mar-31 16:13:01.261 [main] DEBUG nextflow.trace.ReportObserver - Flow completing -- rendering html report
Mar-31 16:13:01.302 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
  [{"cpuUsage":{"mean":91.4,"min":91.4,"q1":91.4,"q2":91.4,"q3":91.4,"max":91.4,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"process":"run_indexcov","mem":{"mean":56840192,"min":56840192,"q1":56840192,"q2":56840192,"q3":56840192,"max":56840192,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"memUsage":{"mean":0.66,"min":0.66,"q1":0.66,"q2":0.66,"q3":0.66,"max":0.66,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"timeUsage":null,"vmem":{"mean":140840960,"min":140840960,"q1":140840960,"q2":140840960,"q3":140840960,"max":140840960,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"reads":{"mean":8431460,"min":8431460,"q1":8431460,"q2":8431460,"q3":8431460,"max":8431460,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"cpu":{"mean":91.4,"min":91.4,"q1":91.4,"q2":91.4,"q3":91.4,"max":91.4,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"time":{"mean":2731,"min":2731,"q1":2731,"q2":2731,"q3":2731,"max":2731,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"},"writes":{"mean":6472897,"min":6472897,"q1":6472897,"q2":6472897,"q3":6472897,"max":6472897,"minLabel":"run_indexcov","maxLabel":"run_indexcov","q1Label":"run_indexcov","q2Label":"run_indexcov","q3Label":"run_indexcov"}},{"cpuUsage":null,"process":"build_covviz_report","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":{"mean":1752,"min":1550,"q1":1651,"q2":1752,"q3":1853,"max":1954,"minLabel":"build_covviz_report","maxLabel":"build_covviz_report","q1Label":"build_covviz_report","q2Label":"build_covviz_report","q3Label":"build_covviz_report"},"writes":null},{"cpuUsage":null,"process":"smoove_call","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":null,"writes":null}]
Mar-31 16:13:02.196 [main] DEBUG nextflow.trace.TimelineObserver - Flow completing -- rendering html timeline
Mar-31 16:13:02.414 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Mar-31 16:13:02.744 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
(base) rcug@hpc-rc08:/ngsssd1/rcug/singularity/smoove-nf$
brwnj commented 4 years ago

This likely won't fix it, but make sure you use a pattern for the bams, like:

--bams '/lager2/rcug/2017/922E/DRAGEN_BAM/*_RG.markdup.calmd.bam'

I also pushed a fix for this line: 

Mar-31 16:12:53.018 [main] WARN  nextflow.script.ScriptBinding - Access to undefined parameter `bed` -- Initialise it to a default value eg. `params.bed = some_value`

I'm actively debugging the workflow with a single bam to see if that's contributing to this error.