I am currently using the following commands to run ipdSummary on our compute server and it is taking some samples 6+ days to run without completing. We have 2TB of RAM and 130 CPUs so I do not believe there is a resource limitation. I have the command set to run on 20 threads and I can see from the log files that for many of the running jobs, about ~15 of 20 threads have exited: "Process KineticWorkerProcess-9 (PID=1919193) done; exiting." However the few remaining processes are struggling along and only using a few CPUs. For example, I have 6 jobs running on 120 total CPUs, but only 47 are in use. During the assembly (Flye) all 120 CPUs were in use so it is not an issue with our queuing system.
Could you look at my ipdSummary command and give some pointers on ways to speed up this process? Any and all help is appreciated here! Thanks
Hello,
I am currently using the following commands to run ipdSummary on our compute server and it is taking some samples 6+ days to run without completing. We have 2TB of RAM and 130 CPUs so I do not believe there is a resource limitation. I have the command set to run on 20 threads and I can see from the log files that for many of the running jobs, about ~15 of 20 threads have exited: "Process KineticWorkerProcess-9 (PID=1919193) done; exiting." However the few remaining processes are struggling along and only using a few CPUs. For example, I have 6 jobs running on 120 total CPUs, but only 47 are in use. During the assembly (Flye) all 120 CPUs were in use so it is not an issue with our queuing system.
Could you look at my ipdSummary command and give some pointers on ways to speed up this process? Any and all help is appreciated here! Thanks