Runtime for prune_graph

[NicMAlexandre]

Hello,

I am running prune_graph on the output file from ngsLD:

prune_graph/target/release/prune_graph --header --in LD.out -n 10 --weight-field "r2" --weight-filter "r2 >= 0.2" --out unlinked.pos

LD.out is 160Gb. This has been running for about 18 hours, is this pretty typical with 10 threads?

[fgvieira]

How many threads is your process really using? From what I have seen, the number of threads used is heavily dependent on the form of the graph.

[NicMAlexandre]

It looks like only 1:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4146427 user 20 0 25.7g 22.2g 2620 R 100.3 4.4 3424:00 prune_graph

[fgvieira]

How many chromosomes/scaffolds do you have?

[NicMAlexandre]

About 20 chromosomes and 20 scaffolds

[NicMAlexandre]

I'm going to try rerunning on each scaffold separately since the output of ngsLD won't be calculating LD between sites on different chromosomes (--max_kb_dist 100).

[NicMAlexandre]

I'm running each chromosome separately now, but the software is still only using 1 thread even under a slurm system. Largest files are 16Gb so hopefully won't take too long.

[fgvieira]

Current parallelization is not optimal since it only works in highly fragmented graphs. I am working on another approach and will release a new version soon (next couple of weeks) that should be at least 3x faster.

[NicMAlexandre]

Oh very cool, I will definitely use it for a couple of other projects.

Quick question:

All of my chromosome output files that I've run separately have data, but the servers are still running a few processes. Does the output file get populated in batches or is the process stalling after completion?

[NicMAlexandre]

Success!