search

ssadedin / bpipe

Bpipe - a tool for running and managing bioinformatics pipelines
http://docs.bpipe.org/
Other
225 stars 57 forks source link

BUG: bpipe parsing command-line memory specification incorrectly #148

Closed gdevenyi closed 9 years ago

gdevenyi commented 9 years ago

I'm calling bpipe like this:

bpipe run -n 8 -m 14GB ../minc-bpipe-library/pipeline.bpipe <files>

The bpipe log file looks like:

      10 bpipe.Config    [1]     INFO    |7:11:39 Merging pipeline config file 
      11 bpipe.Utils     [1]     INFO    |7:11:39 Read user config executed in 0.633 seconds 
      12 bpipe.Runner    [1]     INFO    |7:11:39 Maximum threads specified as 8 
      13 bpipe.Runner    [1]     INFO    |7:11:39 Maximum memory specified as 14GB 
      14 bpipe.Runner    [1]     INFO    |7:11:39 Loading tool database ...  
      15 bpipe.ToolDatabase      [23]    INFO    |7:11:39 Loading tool database from user configuration 
      16 bpipe.Concurrency       [25]    INFO    |7:11:39 Creating thread pool with 8 threads to execute parallel pipelines 
      17 bpipe.NotificationManager       [26]    INFO    |7:11:39 Configuring notifications 
      18 bpipe.NotificationManager       [26]    INFO    |7:11:39 Trying class name file for notification channel file 
      19 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool samtools 
      20 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool bwa 
      21 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool bowtie 
      22 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool fastqc 
      23 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool coverageBed 
      24 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool fastx_clipper 
      25 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool convert2annovar.pl 
      26 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool summarize_annovar.pl 
      27 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool bowtie2 
      28 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool GenomeAnalysisTK 
      29 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool MarkDuplicates 
      30 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool signalp 
      31 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool variant_effect_predictor.pl 
      32 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool igvtools 
      33 bpipe.Tool      [23]    INFO    |7:11:39 Configured meta data about tool ls 
      34 bpipe.Concurrency       [25]    INFO    |7:11:39 Setting maximum memory to 4000 from configuration / command line

Note the following line in the log:

      34 bpipe.Concurrency       [25]    INFO    |7:11:39 Setting maximum memory to 4000 from configuration / command line

Somewhere, bpipe has picked up 4000 as the memory, so my 6GB jobs never run

gdevenyi commented 9 years ago

In comparison, here's running with -m 14000MB

       1 bpipe.Runner    [1]     INFO    |3:49:45 Starting 
       2 bpipe.Runner    [1]     INFO    |3:49:45 OS: Linux (2.6.32-504.3.3.el6.x86_64) Java: 1.8.0_45 Vendor: Oracle Corporation 
       3 bpipe.Runner    [1]     INFO    |3:49:45 Initializing plugins ... 
       4 bpipe.Config    [1]     INFO    |3:49:45 No plugins directory found: /scratch/m/mchakrav/gdevenyi/bpipe-adni-test/output/.bpipe/plugins 
       5 bpipe.Runner    [1]     INFO    |3:49:45 Reading user config ...  
       6 bpipe.Config    [1]     INFO    |3:49:45 Reading Bpipe configuration from /scratch/m/mchakrav/gdevenyi/bpipe-adni-test/output/../minc-bpipe-library/bpipe.config 
       7 bpipe.Config    [1]     INFO    |3:49:45 No local configuration file found 
       8 bpipe.Utils     [22]    INFO    |3:49:45 Read config from /scratch/m/mchakrav/gdevenyi/bpipe-adni-test/output/../minc-bpipe-library/bpipe.config executed in 0.233 seconds 
       9 bpipe.Utils     [21]    INFO    |3:49:45 Read config from /project/m/mchakrav/quarantine/bpipe/0.9.8.7/build/bpipe-0.9.8.7/bin/../bpipe.config executed in 0.276 seconds 
      10 bpipe.Config    [1]     INFO    |3:49:45 Merging pipeline config file 
      11 bpipe.Utils     [1]     INFO    |3:49:45 Read user config executed in 0.580 seconds 
      12 bpipe.Runner    [1]     INFO    |3:49:45 Maximum threads specified as 8 
      13 bpipe.Runner    [1]     INFO    |3:49:45 Maximum memory specified as 14000MB 
      14 bpipe.Runner    [1]     INFO    |3:49:45 Loading tool database ...  
      15 bpipe.ToolDatabase      [23]    INFO    |3:49:45 Loading tool database from user configuration 
      16 bpipe.NotificationManager       [26]    INFO    |3:49:45 Configuring notifications 
      17 bpipe.Concurrency       [25]    INFO    |3:49:45 Creating thread pool with 8 threads to execute parallel pipelines 
      18 bpipe.NotificationManager       [26]    INFO    |3:49:45 Trying class name file for notification channel file 
      19 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool samtools 
      20 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool bwa 
      21 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool bowtie 
      22 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool fastqc 
      23 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool coverageBed 
      24 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool fastx_clipper 
      25 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool convert2annovar.pl 
      26 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool summarize_annovar.pl 
      27 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool bowtie2 
      28 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool GenomeAnalysisTK 
      29 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool MarkDuplicates 
      30 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool signalp 
      31 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool variant_effect_predictor.pl 
      32 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool igvtools 
      33 bpipe.Tool      [23]    INFO    |3:49:45 Configured meta data about tool ls 
      34 bpipe.Runner    [1]     INFO    |3:49:45 No CLI parameters specified 
      35 bpipe.NotificationManager       [26]    INFO    |3:49:46 Unable to create notification channel using class file: java.lang.ClassNotFoundException: file 
      36 bpipe.NotificationManager       [26]    INFO    |3:49:46 Trying class name bpipe.FILENotificationChannel for notification channel file 
      37 bpipe.NotificationManager       [26]    INFO    |3:49:46 Unable to create notification channel using class bpipe.FILENotificationChannel: java.lang.ClassNotFoundException: bpipe.FILENotificationChannel 
      38 bpipe.NotificationManager       [26]    INFO    |3:49:46 Trying class name bpipe.file for notification channel file 
      39 bpipe.NotificationManager       [26]    INFO    |3:49:46 Unable to create notification channel using class bpipe.file: java.lang.ClassNotFoundException: bpipe.file 
      40 bpipe.NotificationManager       [26]    INFO    |3:49:46 Trying class name bpipe.FileNotificationChannel for notification channel file 
      41 bpipe.NotificationManager       [26]    INFO    |3:49:46 Successfully created notification channel using class bpipe.FileNotificationChannel 
      42 bpipe.Runner    [1]     INFO    |3:49:46 Parsing script ...  
      43 bpipe.Runner    [1]     INFO    |3:49:46 Run ...  
      44 bpipe.Pipeline  [1]     INFO    |3:49:46 Evaluating library file /scratch/m/mchakrav/gdevenyi/bpipe-adni-test/minc-bpipe-library/minc-library.bpipe 
      45 bpipe.PipelineCategory  [1]     INFO    |3:49:46 Found closure variable mincconvert 
      46 bpipe.PipelineCategory  [1]     INFO    |3:49:46 Found closure variable n3correct 
      47 bpipe.PipelineCategory  [1]     INFO    |3:49:46 Found closure variable nlm_denoise
gdevenyi commented 9 years ago

Confirmed in #149 that command line parsing of MB and GB are both broken

ssadedin commented 9 years ago

Thanks - parsing issues fixed in 2c65801.