galaxyproject / usegalaxy-tools

usegalaxy.* common tools
11 stars 52 forks source link

Request: increase memory allocation for mashmap #833

Open jennaj opened 1 week ago

jennaj commented 1 week ago

toolshed.g2.bx.psu.edu/repos/iuc/mashmap/mashmap/3.1.3+galaxy0

Another example job from Ross @fubar2 that fails at ORG but runs Ok at EU (same inputs/params/tool version). Seems to have similar default mem/cores at both. Maybe rolled over to higher mem at EU?

fubar2 commented 1 week ago

@natefoo @mvdbeek @jennaj: tl;dr Mashmap is tricky as RAM useage depends heavily on input and parameters - 4GB - 60GB so there may be a good argument for setting boundaries on the % identity and match length to keep the RAM usage sane

Have tried a couple of sets of parameters with VGP sized inputs. The size of all the fasta matters, but a low percentage identity and a short match length will result in much, much higher RAM use.

For example, this with about 6GB of fasta @ 90% identity for 1000nt which is a relatively easy criterion to match so takes up a lot of RAM and almost 10X time: image

whereas with the same fasta @ a much more stringent match requirement of 95% identity over 5000nt uses much less: image

fubar2 commented 4 days ago

@mvdbeek: With the current

Cores Allocated     8
Memory Allocated (MB)   28672

seeing OOM failures for 1k/93% and 5k/85% (MashMap defaults) settings on typical VGP inputs. Four other combinations and inputs work fine. https://usegalaxy.org/u/fubar/h/treevalgalaugtwohaps-hg002