question for continuous checkpoints and pbs executor

[biowang]

Dear sir. I encounter two question, as follow

_1_ . I don`t find the "Continuous checkpoints" function. processes still Submitted after being sucessfully executed.

=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W  ~  version 0.12.0
=message= [warm up] executor > local
=message= [1b/f5ade3] Submitted process > validate_utr (1)
=message= [ed/0a1fa0] Submitted process > validate_mir (1)
=message= [3d/9b255f] Submitted process > cor_mir_utr (3)
=message= [fa/3a387a] Submitted process > cor_mir_utr (1)
=message= ...
=message= ...
=message= [88/8e6df7] Submitted process > cor_mir_utr (104)
=message= Result saved to file: /llwang-works/data/ceRNA/example_nextflow/result.txt
...
=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W  ~  version 0.12.0
=message= [warm up] executor > local
=message= [1b/f5ade3] Submitted process > validate_utr (1)
=message= [ed/0a1fa0] Submitted process > validate_mir (1)
=message= [3d/9b255f] Submitted process > cor_mir_utr (3)
=message= [fa/3a387a] Submitted process > cor_mir_utr (1)
=message= [e6/8b257c] Submitted process > cor_mir_utr (7)
=message= [95/dbd725] Submitted process > cor_mir_utr (5)

_2_ . using pbs executor , the processing stop at the last process...

=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W  ~  version 0.12.0
=message= [warm up] executor > pbs
=message= [ae/0543aa] Submitted process > validate_utr (1)
=message= [a7/4482ba] Submitted process > validate_mir (1)
=message= [51/abc5ae] Submitted process > cor_mir_utr (4)
=message= [b8/a1b37e] Submitted process > cor_mir_utr (3)
=message= [70/f6827c] Submitted process > cor_mir_utr (9)
=message= [16/af8b6e] Submitted process > cor_mir_utr (103)
=message= [b4/46e6e4] Submitted process > cor_mir_utr (104)

my script as follow

  1 // parameters
  2 params.mirfile     = "/llwang-works/data/ceRNA/example_nextflow/data/expr/mir.expr"
  3 params.utrfile     = "/llwang-works/data/ceRNA/example_nextflow/data/expr/utr.expr"
  4 params.chunkSize   = 200
  5
  6 params.out         = "/llwang-works/data/ceRNA/example_nextflow/result.txt"
  7
  8
  9 // Channels
 10 mirCh = Channel.fromPath(params.mirfile)
 11 utrCh = Channel.fromPath(params.utrfile)
 12
 13
 14 // validate utr sequence
 15 process validate_utr {
 16     input:
 17     file 'source' from utrCh
 18
 19     output:
 20     file 'valid' into utrValCh
 21
 22     """
 23     perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
 24     """
 25 }
 26
 27
 28 // validate microRNA sequence
 29 process validate_mir {
 30     input:
 31     file 'source' from mirCh
 32
 33     output:
 34     file 'valid' into mirValCh
 35
 36     """
 37     perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
 38     """
 39 }
 40
 41
 42 utrValSplitCh = utrValCh.splitText( by: params.chunkSize )
 43
 44
 45 process cor_mir_utr {
 46     input:
 47     file 'utr' from utrValSplitCh
 48     file 'mir'  from mirValCh.first()
 49
 50     output:
 51     file 'corel' into corCh
 52
 53     """
 54     Rscript /llwang-works/data/ceRNA/example_nextflow/scripts/cor.R utr mir 0.05 0.7 corel
 55     """
 56 }
 57
 58 corCh
 59   .collectFile(name: file(params.out))
 60   .subscribe {
 61      println "Result saved to file: $it"
 62    }
 63

[pditommaso]

Please use the discussion group https://groups.google.com/forum/#!forum/nextflow

[biowang]

I am sorry , I am in China. I can`t access the google website. can you get me answer of this question? Thank you

[pditommaso]

I see, OK. In order to skip the executions of jobs already processed you need to add the -resume option to nextflow command line when you run your script.

The second question is more difficult. Can you upload the .nextflow.log file ?

[biowang]

I provide you three links for pbs.log.

1. https://www.dropbox.com/s/5yfjkbsjlcm55y5/pbs.log?dl=0 2.http://pan.baidu.com/s/1kTFZO3t
2. http://www.evernote.com/l/AICB4bLRBTlNCYHunSD7wV9fDAGH0etuakA/

[pditommaso]

Likely, there's a problem while submitting the job to the cluster, but the pipeline hung due to a bug that a resolved in the latest version. Please update to nextflow 0.12.3

However, have you installed PBS in a virtual machine ? it looks odd to me that the the IP address in your log file is 127.0.0.1

[biowang]

I don`t have a virtual machine. my pipeline was submitted on a node of pbs cluster.

it update to nextflow 0.12.3 , but some pipeline failed submitted to pbs cluster. please see the issue below: https://github.com/nextflow-io/nextflow/issues/32#issuecomment-75982648

[pditommaso]

The problem was that the user was not using a shared file system when using the PBS executor (see comment on #32)