search

zamboni-lab / SLAW

Scalable and self-optimizing processing workflow for untargeted LC-MS
GNU General Public License v2.0
25 stars 3 forks source link

Cgroup mem limit exceeded #33

Closed song-sbio closed 9 months ago

song-sbio commented 11 months ago

I'm trying to run slaw for peak picking for 676 samples in mzML format. It was working fine, but I encountered an error after the annotation stage and the datamatrices file is empty. Would be grateful if you can advise how I can fix the issue. Below is the error message. Thank you!

2023-07-19|10:15:37|INFO: Total memory available: 328350 and 256 cores. The workflow will use 1287 Mb by core on 255 cores. 2023-07-19|10:15:37|INFO: Guessing polarity from file:Bat6_a13.17.4.2.mzML 2023-07-19|10:15:41|INFO: Polarity detected: negative 2023-07-19|10:15:43|INFO: STEP: initialisation TOTAL_TIME:5.44s LAST_STEP:5.44s 2023-07-19|10:15:44|INFO: 676 peakpicking added 2023-07-20|06:25:31|INFO: MS2 extraction finished 2023-07-20|06:25:31|INFO: Starting peaktable filtration 2023-07-20|06:25:36|INFO: Done peaktables filtration 2023-07-20|06:25:36|INFO: STEP: peakpicking TOTAL_TIME:72598.54s LAST_STEP:72593.10s 2023-07-20|06:25:36|INFO: Aligning 2023-07-20|06:25:43|INFO: Filtering 2023-07-20|06:25:44|INFO: Extracting consensus MS-MS spectra 2023-07-20|06:25:47|INFO: Alignment finished 2023-07-20|06:25:47|INFO: STEP: alignment TOTAL_TIME:72610.08s LAST_STEP:11.54s 2023-07-20|06:25:47|INFO: Gap filling and isotopic pattern extraction finished. 2023-07-20|06:25:47|INFO: STEP: gap-filling TOTAL_TIME:72610.09s LAST_STEP:0.00s 2023-07-20|06:25:47|INFO: Annotation finished 2023-07-20|06:25:47|INFO: STEP: annotation TOTAL_TIME:72610.09s LAST_STEP:0.01s Cgroup mem limit exceeded: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=2010946.pbs101,mems_allowed=0-7,oom_memcg=/pbs_jobs.service/jobid/2010946.pbs101,task_memcg=/pbs_jobs.service/jobid/2010946.pbs101,task=R,pid=2522386,uid=28009

song-sbio commented 10 months ago

I tried to run debug code as well (I'm using SLAW as a singularity container). Attached is the debug report. It seems the RAM was exceeded at the annotation stage. slaw_no_opt.txt Would be grateful if you can advise. Thank you!

htmonkey commented 10 months ago

1) Pls run the container with -e logging=debug to get the full slaw log 2) Use less cores. Running 256 cores for 676 is unnecessary. Reduce it to cores, and you'll still get fast output. 3) Run the job locally. You can run the optimization on the cluster, and then run the alignment etc locally with the parameters.txt file that you obtain from the cluster.

song-sbio commented 9 months ago
  1. Pls run the container with -e logging=debug to get the full slaw log
  2. Use less cores. Running 256 cores for 676 is unnecessary. Reduce it to cores, and you'll still get fast output.
  3. Run the job locally. You can run the optimization on the cluster, and then run the alignment etc locally with the parameters.txt file that you obtain from the cluster.

Hi, Thank you for pointing out the issue that I was still using too many cores. Previously I tried to reduce the number of cores by setting it in the PBS instruction sheet, but it seemed that it could not reduce the number of cores used. After I used SINGULARITYENV_LOGGING=DEBUG SINGULARITYENV_MEMORY=2000 SINGULARITYENV_NCORES=24, I managed to reduce the number of cores used and the codes ran successfully. Thank you very much for developing the software and answering my questions.