Closed metasoarous closed 6 years ago
Ugh. Do you know about nw_distance
from the Newick utilities? It would require an extra parsing step, but it might be worth trying out on your big trees.
Good idea! I'll give that a whirl and see how it does on memory. Thanks.
I may have mostly resolved this now by submitting the bigger jobs to the largenode partition. Turns out I wasn't doing this right initially, because of other constraints that have to be added to the srun invocation for those nodes to be available. There are still a few jobs failing, but I'm having trouble finding them because the scons build log overflowed my tmux pane buffer :-/ So still some hunting to fully resolve this, but we've at least got the laura-mb dataset built now.
As clarified in #226, the last remaining issues aren't actually with the minadcl_clusters.py
but upstream of that with the invocation of process_partis.py
. All of the clusters for which the process_partis.py
issue isn't hitting us seem to be making it through minadcl_clusters.py
now as a result of 653b20b (push pending).
It seems that we're overflowing as we create the distance matrix. I've improved the situation some by exercising more control over how we submit jobs, setting memory requirements and exclusive access for these jobs. With more cluster tinkering, I may be able to get this to work (been in touch with scicomp about it).