out of memory processing when operating on 500+ files

[GoogleCodeExporter]

Running a simple bpipe script like

get_quality = {
   exec """
   qc.pl -q -s 0 -i ${input} &&
   touch $output
   """
  forward input
}

summarise = {
  produce("summary.txt"){
    exec """
      summarize.py .
    """
  }
}

Bpipe.run {
  "%" * [ get_quality ] + summarise
}

operating on 500+ files I get OutOfMemory exception even though I set 
: ${MAX_JAVA_MEM:="1024m"}
in bin/bpipe.

I know my example is poorly described, I just wanted to hear what you typically 
set MAX_JAVA_MEM too, if it is normal that bpipe requires a lot of memory like 
this?

...and btw, thanks for an awesome project we are getting more and more into 
bpipe here in my office!

Original issue reported on code.google.com by henning....@gmail.com on 27 May 2014 at 8:02