computational-metabolomics / metfrag-galaxy

Metfrag for Galaxy
GNU General Public License v3.0
3 stars 3 forks source link

let metfrag detect out of memory condition #12

Closed bernt-matthias closed 5 years ago

bernt-matthias commented 5 years ago

When submitting with to less memory I get:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.UnsatisfiedLinkError: /gpfs1/data/galaxy_server/galaxy/database/dependencies/_conda/envs/__metfrag@2.4.2/lib/libnio.so: /gpfs1/data/galaxy_server/galaxy/database/dependencies/_conda/envs/__metfrag@2.4.2/lib/libnio.so: failed to map segment from shared object: Cannot allocate memory

in stderr.

Maybe detect Cannot allocate memory as out of memory error, then Galaxy can react on this.

bernt-matthias commented 5 years ago

ping @chufz

Tomnl commented 5 years ago

OK - thanks for the issue.

We have experienced memory issues before and dynamically assigned the memory based on the maximum number of peaks with a single spectra within the MSP.

e.g.

 <destination id="destination_metfrag" runner="dynamic">
            <param id="type">python</param>
            <param id="function">metfrag_msp_peak_count</param>
  </destination>

@RJMW when you get a chance could you copy and paste the python function we ended up using please

bernt-matthias commented 5 years ago

OK. This is also an option.

What I would suggest is to add

<stdio>
    <regex match="Cannot allocate memory"
           source="stderr"
           level="fatal_oom"
           description="Out of memory error occurred" />
</stdio>

then Galaxy can resubmit to a destination with more memory.

Tomnl commented 5 years ago

This is now implemented.

For reference, if a dynamic runner is to be assigned the function needs to be added to galaxy/lib/galaxy/jobs/rule/destinations_input_type.py e.g. a function called metfrag_msp_peak_count needs to be in `destinations_input_type.py that checks for msp peak count and then assigns memory.

At our local bham galaxy instance we currently check for precursor m/z size to assign levels of memory (However, I think there is probably a better predictor of memory usage that the precursor m/z or number of peaks)

bernt-matthias commented 5 years ago

Wonderful. Just installed the released version on our local Galaxy (ping @chufz).