search

metagenomics / MeRaGENE

MIT License
4 stars 2 forks source link

Remove local execution #129

Closed pbelmann closed 8 years ago

pbelmann commented 9 years ago

Since the local execution of processes is just a quick fix we should remove it for the next 0.2.0 milestone.

I suggest removing the 'local' execution in all processes (except the process discussed in #91) and limit the parallel execution of the tasks by using the maxForks parameter. This gives a scheduler enough time to schedule tasks even if they are fast.

bosterholz commented 9 years ago

maxFork sounds like a good/safe way. If you execute the pipeline just now with big data you get this: 

[ad/302097] Submitted process > hmmFolderScan (101)
[f2/1359a5] Submitted process > hmmFolderScan (156)
ERROR ~ Error executing process > 'hmmFolderScan'

Caused by:
  The thread pool executor cannot run the task. The upper limit of the thread pool size has probably been reached. Current pool size: 1000 Maximum pool size: 1000

 -- Check '.nextflow.log' file for details
[ad/628c21] Submitted process > hmmFolderScan (7)
[9c/162681] Submitted process > hmmFolderScan (111)
[c7/f0c753] Submitted process > hmmFolderScan (39)
[8d/1c2047] Submitted process > hmmFolderScan (43)
[21/0101a4] Submitted process > hmmFolderScan (121)
[94/8457db] Submitted process > hmmFolderScan (17)
[ca/eec81e] Submitted process > hmmFolderScan (545)
[cc/60ac2d] Submitted process > hmmFolderScan (53)
[18/5b5df0] Submitted process > hmmFolderScan (122)
[3d/edbbf1] Submitted process > hmmFolderScan (77)
[ab/dc5f14] Submitted process > hmmFolderScan (97)
[bf/76becd] Submitted process > hmmFolderScan (161)
WARN: Killing pending processes (20)
[65/1e8641] Submitted process > hmmFolderScan (13)
pbelmann commented 9 years ago

Are you running it with the standard or local executor?

bosterholz commented 9 years ago

standard, nothing changed except for the right paths for a "big"-run

pbelmann commented 9 years ago

Ok it's my fault. I noticed that maxforks can just be used for threading and not for qsub like tasks. I think we should remove it for now. It is not the solution for our problem.

pbelmann commented 9 years ago

We have to find another way to remove local execution.

bosterholz commented 9 years ago

No, I think you made the right discovery. But you used it local with maxFork=6000 which overwrote my global maxFork=500.I restarted it without the maxFork=6000. It is still running without issues. Keep your fingers crossed :pray:

pbelmann commented 9 years ago

Even if it works it is not the same as the qsub parameter -tc max_running_task. The nextflow documentation states that it is meant for restricting parallel tasks on a local run: http://www.nextflow.io/docs/latest/process.html#maxforks

Since your run works without restricting maximum parallel taks we should think about leaving everything as it is without changing the processes. For this we should investigate the trace file again.