MANTA is often killed after 6 hours before it completes

[davidcotter]

Hello

I am running your great tool but quite often I get this message in the MANTA log:

[2019-02-04T13:24:29.666749Z] [6a4c73e58738] [119_1] [TaskManager] Completed command task: 'generateCandidateSV_0204' launched from master workflow
[2019-02-04T13:24:29.694216Z] [6a4c73e58738] [119_1] [TaskManager] Launching command task: 'generateCandidateSV_0169' from master workflow
[2019-02-04T13:24:29.745750Z] [6a4c73e58738] [119_1] [TaskRunner:generateCandidateSV_0169] Task initiated on local node
[2019-02-04T13:24:40.937949Z] [6a4c73e58738] [119_1] [WorkflowRunner] [ERROR] Received termination signal, shutting down running tasks...

I notice that this is exactly 6 hours after it starts and I seems to remember some timeout in the code or somewhere else but can't find it now. I am running this on a decent machine with 32 CPUs and 240G of RAM and it does seems to be near the end of the process. Is there a timeout and can I increase it?

Many thanks.

[davidcotter]

My friend just pointed out the following line

 timeout 6h runManta 1> /home/dnanexus/out/log_files/manta_logs/"${prefix}".manta.stdout.log 2> /home/dnanexus/out/log_files/manta_logs/"${prefix}".manta.stderr.log &

I suggest increasing this limit or making it a parameter. I think my machine that I'm running on is OK but the nfs drive may be slow which might be why the whole thing is taking more than 6 hours.

[davidcotter]

I changed all the 6 hour timeouts to 18 hours and found that Manta took 6 hrs 45 minutes to complete - but delly seemed to go on forever on Chr 2 so I killed it in the end. The thing about delly is that if it times out you still get all the calls up to that point but with Manta if it times out all calls are lost.

I think increasing Manta to 8 hours might be a good idea.

[slzarate]

Hi @davidcotter, thanks for using Parliament2! As you observed, Manta does in fact have a 6-hour timeout based on our own benchmarks. However, your suggestion of adding a customizable timeout for Manta is a good one, and I will add that to the list of issues to fix in the next version. Thanks

[MaestSi]

Hi, was the issue fixed in the 0.1.10 version? I found that Parliament2 did not produce any Manta vcf output, and in the .manta.stderr.log file I found error message:

[ERROR] Received termination signal, shutting down running tasks...

Is it easy to modify the timeout parameter using the docker image? Thanks, Simone

[davidcotter]

A quick way to modifiy parliament2.sh

https://github.com/dnanexus/parliament2/blob/master/dx_app_code/parliament2/parliament2.py

search for timeout 6h runManta .. and replace 6h with the number of hours you think: I went as high as 18 to be sure it finished.

Then build a new docker image with the following Dockerfile

FROM dnanexus/parliament2 COPY parliament2.sh /home/dnanexus/parliament2.sh ENTRYPOINT ["python","/home/dnanexus/parliament2.py"]

and parliament2.sh in a directory use the command

docker build . -t myparliament2

then you can run the new docker image

docker run -it myparliament2

[MaestSi]

Thank you very much!