search

ssadedin / bpipe

Bpipe - a tool for running and managing bioinformatics pipelines
http://docs.bpipe.org/
Other
225 stars 57 forks source link

"Running Pipeline Stages in Parallel" example not working #173

Closed BenKaehler closed 8 years ago

BenKaehler commented 8 years ago

Dear bpipe,

I'm trying to run an example in the "Running Pipeline Stages in Parallel", but it doesn't appear to work. I'm using the bpipe-0.9.9 release on a Mac.

Firstly, the "nice_to_see_you" stage isn't defined anywhere, so I've removed it from the script, which now looks like this:

hello = {
  exec "echo Hello | cat - $input.txt > $output.txt"
}
world = {
  exec "echo World | cat $input.txt - > $output.txt"
}
run { "input_%.txt" ** [ hello + world ] }

I've touched two local files, input_1.txt and input_2.txt.

My session follows. Any help gratefully accepted.

Thanks, Ben

$ bpipe run test.pipe input_*.txt
====================================================================================================
|                              Starting Pipeline at 2016-03-07 16:04                               |
====================================================================================================
=========================================== Bpipe Error ============================================

An error occurred executing your pipeline:

No signature of method: java.lang.String.power() is applicable for argument types: (java.util.ArrayList) values: [[bpipe.DefinePipelineCategory$_plus_closure3@4686afc2]]
Possible solutions: toSet(), toSet(), intern(), size(), size(), every()

Please see the details below for more information.

========================================== Error Details ===========================================

groovy.lang.MissingMethodException: No signature of method: java.lang.String.power() is applicable for argument types: (java.util.ArrayList) values: [[bpipe.DefinePipelineCategory$_plus_closure3@4686afc2]]
Possible solutions: toSet(), toSet(), intern(), size(), size(), every()
    at test.pipe$_run_closure3.doCall(test.pipe.groovy:7)
    at test.pipe$_run_closure3.doCall(test.pipe.groovy)
    at bpipe.Pipeline$_diagram_closure21.doCall(Pipeline.groovy:1095)
    at bpipe.Pipeline$_diagram_closure21.doCall(Pipeline.groovy)
    at bpipe.Pipeline.diagram(Pipeline.groovy:1094)
    at bpipe.Pipeline$diagram$4.callCurrent(Unknown Source)
    at bpipe.Pipeline.execute(Pipeline.groovy:603)
    at bpipe.Pipeline.execute(Pipeline.groovy)
    at bpipe.Pipeline.this$2$execute(Pipeline.groovy)
    at bpipe.Pipeline$this$2$execute$1.call(Unknown Source)
    at bpipe.Pipeline.run(Pipeline.groovy:483)
    at bpipe.Pipeline$run.callStatic(Unknown Source)
    at bpipe.Pipeline.run(Pipeline.groovy:443)
    at test.pipe.run(test.pipe.groovy:7)
    at test.pipe$run.call(Unknown Source)
    at bpipe.Runner.main(Runner.groovy:414)

====================================================================================================

More details about why this error occurred may be available in the full log file .bpipe/bpipe.log

/Applications/Bpipe/bin/bpipe: line 744:  8069 Terminated: 15          ( tail -f $LOGFILE | sed -l "$TAIL_PATTERN" )
$ cat .bpipe/bpipe.log 
bpipe.Runner    [1] INFO    |4:04:27 Starting 
bpipe.Runner    [1] INFO    |4:04:27 OS: Mac OS X (10.11.1) Java: 1.8.0_66 Vendor: Oracle Corporation 
bpipe.Runner    [1] INFO    |4:04:27 Initializing plugins ... 
bpipe.Config    [1] INFO    |4:04:27 No plugins directory found: /Users/ben/Projects/bpipe-test/.bpipe/plugins 
bpipe.Runner    [1] INFO    |4:04:27 Reading user config ...  
bpipe.Config    [1] INFO    |4:04:27 No configuration file found in same dir as pipeline file /Users/ben/Projects/bpipe-test/bpipe.config 
bpipe.Config    [1] INFO    |4:04:27 No local configuration file found 
bpipe.Utils [13]    INFO    |4:04:28 Read config from /Applications/Bpipe/bin/../bpipe.config executed in 0.211 seconds 
bpipe.Utils [1] INFO    |4:04:28 Read user config executed in 0.384 seconds 
bpipe.Runner    [1] INFO    |4:04:28 Loading tool database ...  
bpipe.ToolDatabase  [14]    INFO    |4:04:28 Loading tool database from user configuration 
bpipe.Dependencies  [18]    INFO    |4:04:28 No cached output graph (outputGraph.ser) available: will be computed from property files 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Configuring notifications 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Trying class name file for notification channel file 
bpipe.Concurrency   [16]    INFO    |4:04:28 Creating thread pool with 64 threads to execute parallel pipelines 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool samtools 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool bwa 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool bowtie 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool fastqc 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool coverageBed 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool fastx_clipper 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool convert2annovar.pl 
bpipe.Runner    [1] INFO    |4:04:28 No CLI parameters specified 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool summarize_annovar.pl 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool bowtie2 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool GenomeAnalysisTK 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool MarkDuplicates 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool signalp 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool variant_effect_predictor.pl 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool igvtools 
bpipe.Tool  [14]    INFO    |4:04:28 Configured meta data about tool ls 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Unable to create notification channel using class file: java.lang.ClassNotFoundException: file 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Trying class name bpipe.FILENotificationChannel for notification channel file 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Unable to create notification channel using class bpipe.FILENotificationChannel: java.lang.ClassNotFoundException: bpipe.FILENotificationChannel 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Trying class name bpipe.file for notification channel file 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Unable to create notification channel using class bpipe.file: java.lang.ClassNotFoundException: bpipe.file 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Trying class name bpipe.FileNotificationChannel for notification channel file 
bpipe.NotificationManager   [17]    INFO    |4:04:28 Successfully created notification channel using class bpipe.FileNotificationChannel 
bpipe.Runner    [1] INFO    |4:04:28 Parsing script ...  
bpipe.Runner    [1] INFO    |4:04:28 Run ...  
bpipe.Pipeline  [1] INFO    |4:04:28 Running with input [input_1.txt, input_2.txt] 
bpipe.PipelineCategory  [1] INFO    |4:04:28 Found closure variable hello 
bpipe.PipelineCategory  [1] INFO    |4:04:28 Found closure variable world 
bpipe.Pipeline  [1] WARNING |4:04:28 Pipeline folder /Users/ben/bpipes could not be found 
bpipe.Pipeline  [1] INFO    |4:04:28 Loaded external reference: args 
bpipe.Pipeline  [1] INFO    |4:04:28 External reference args is overridden by local reference 
bpipe.Pipeline  [1] INFO    |4:04:28 Loaded external reference: hello 
bpipe.Pipeline  [1] INFO    |4:04:28 External reference hello is overridden by local reference 
bpipe.Pipeline  [1] INFO    |4:04:28 Loaded external reference: world 
bpipe.Pipeline  [1] INFO    |4:04:28 External reference world is overridden by local reference 
bpipe.Runner    [1] SEVERE  |4:04:28 Reporting exception to user:  
bpipe.Runner    [1] SEVERE  |4:04:28 Reporting exception to user 
Exception:
groovy.lang.MissingMethodException: No signature of method: java.lang.String.power() is applicable for argument types: (java.util.ArrayList) values: [[bpipe.DefinePipelineCategory$_plus_closure3@4686afc2]]
Possible solutions: toSet(), toSet(), intern(), size(), size(), every()
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at script14573270684181489535454$_run_closure3.doCall(script14573270684181489535454.groovy:7)
    at script14573270684181489535454$_run_closure3.doCall(script14573270684181489535454.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
    at bpipe.Pipeline$_diagram_closure21.doCall(Pipeline.groovy:1095)
    at bpipe.Pipeline$_diagram_closure21.doCall(Pipeline.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
    at groovy.lang.Closure.call(Closure.java:423)
    at groovy.lang.Closure.call(Closure.java:417)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:109)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.access$400(GroovyCategorySupport.java:65)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:249)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.use(DefaultGroovyMethods.java:400)
    at org.codehaus.groovy.runtime.dgm$612.invoke(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:248)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
    at bpipe.Pipeline.diagram(Pipeline.groovy:1094)
    at bpipe.Pipeline$diagram$4.callCurrent(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
    at bpipe.Pipeline.execute(Pipeline.groovy:603)
    at bpipe.Pipeline.execute(Pipeline.groovy)
    at bpipe.Pipeline.this$2$execute(Pipeline.groovy)
    at bpipe.Pipeline$this$2$execute$1.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
    at bpipe.Pipeline.run(Pipeline.groovy:483)
    at bpipe.Pipeline$run.callStatic(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:53)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:157)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:173)
    at bpipe.Pipeline.run(Pipeline.groovy:443)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1339)
    at org.codehaus.groovy.runtime.callsite.StaticMetaClassSite.callStatic(StaticMetaClassSite.java:62)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:53)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:157)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:165)
    at script14573270684181489535454.run(script14573270684181489535454.groovy:7)
    at script14573270684181489535454$run.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
    at bpipe.Runner.main(Runner.groovy:414)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:106)
    at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:128)
ssadedin commented 8 years ago

Hi @BenKaehler, this looks like a problem in the documentation that must have occurred when we ported it over from Google Code. The * character should only appear singly, not doubly in those expressions:

run { "input_%.txt" * [ hello + world ] }

Fixed by da18ed1.

BenKaehler commented 8 years ago

Thanks @ssadedin! That's fixed my immediate problem.

You may wish to have another look at that doc, however, because it's pretty mangled. As I said above, nice_to_see_you is missing altogether. Also, ** appears in the text, and several of the examples look badly broken (at least to a non-groovy speaker like me). For example:

Bpipe.run {
  hello + [}
ssadedin commented 8 years ago

Thanks @BenKaehler - Is that last example on the same page? I couldn't find it.

BenKaehler commented 8 years ago

Sorry @ssadedin, you're right, that example appears to have been fixed in the repo.

That example comes from the online docs (http://bpipe-test-documentation.readthedocs.org/en/latest/Language/ParallelTasks/), and then I somehow ended up looking at looking at a forked version (https://github.com/lonsbio/bpipe/blob/master/docs/Language/ParallelTasks.md).