esolares
commented
4 years ago Hi,
It sounds like some of the threads may have crashed and python is waiting for them to finish, but can't because of out of memory issues. By the way, minimap2 paf files seem to take up less ram when running HapSolo. One thing to check is to see how many lines are in your psl file and how large is the file? can you do a ls -lh *.psl Also, how many contigs are in your contig assembly, and what is the current N50? It sounds like HapSolo is being overwhelmed by the number of alignment hits it's getting. Did you do a minimum length filter also?
With respect to the number of iterations, it is currently set to number of iterations per threads. I could add an option say -N that would take the total number of iterations and divide it by the total number of threads and take its ceiling. I understand now that perhaps -n
Thank you,
Edwin
sivico26
Hi again @esolares,
After preparing both blat and Busco inputs, I runned Hapsolo for my genome. Here is the command I used:
/home/svillanu/Programs/HapSolo/hapsolo.py -i $genome -p inputs/all_blat.psl -b inputs/busco -t $threads -n 10000 -B 5The first abnormality arose when the program was running longer than expected (according to the manuscript minutes or hours, I am talking about days here). At first, I thought it was because I increased the number of iterations but then other things arouse as well.
I am used to monitoring new programs to see how they behave. I noticed Hapsolo is quite memory-greedy and even consumed all the memory of my big-memory node going to swap and filling swap as well (378 Gb RAM + 64 Gb Swap). At that point it, this error was thrown to Stderr: "OSError: [Errno 12] Cannot allocate memory".
I thought It would die after that. Surprisingly, it did not. It freed around 5% of RAM and continue to run okay (using 95 or 96 % of RAM) almost until the end. I think the memory overflow occurred between iteration 1000 and 2000, but not sure, it was at the beginning though.
I say almost because in the final days the process was zombie dead, just a few iterations before completion. The load of the node was literally 0 since then. Not sure when that happened though, it was at some point between the 4th and 6th day. Here are the final lines of the stdout log:
The interruption of the last line is weird, but that is how it is in the log. Not sure why. It does not seem like a python thing.
The last thing I found weird it the "CostFxnDelta: 0.0". If I got it right, Hapsolo makes an optimization process and I interpret this variable as the delta of the optimization function between iterations. It is weird that it does not change at all, although the other values change with the iterations. Just to be clear, CostFxnDelta is 0.0 since the beginning (literally since iteration 0), and apparently it never got any different.
After all this, I thought I made a mistake providing the inputs. But the Blat psl is okay and as I understand the input for Busco is the parent directory where the busco### directories are located, is that correct?
The last question that I have is what are the threads doing? Because by the log it seemed like every thread is doing N iterations doing in my case around 240000 iterations, which sound like overkill. How is the sensible way of using them?
Worth to mention that by the logs, both the Busco and Blat inputs seem to be okay.
I really appreciate your thoughts on all this mess.