PrincetonUniversity / DP_GP_cluster

BSD 3-Clause "New" or "Revised" License
73 stars 25 forks source link

Script stalling with --check-convergence option #14

Open ak-watson opened 5 years ago

ak-watson commented 5 years ago

HI,

I have been using DP_GP to cluster RNAseq data across 6 time points for ~1800 genes.

I have been using the --true_times and --check_convergence options.

While trying to run the program I noticed that the script would frequently stall part way through the run. The process would continue to run on 100% of one CPU but with no further output (the maximum time I left it to see if it would recover was 12h). Killing the job and rerunning it meant I could eventually get a result for 1000 and 2000 max iterations, without reaching convergence, but I could not get a clean run at 3000 iterations. Removing the --check-convergence option seems to have resolved the problem.

This may be a bug with the --check-convergence option itself, I'm not sure.

Is there an inbuilt method to check convergence after running the program that could be used as a work around?

Thanks in advance, Andrew