Confusing n procs status messages

[scharch]

I consistently get messages like running 9 procs for 108999 seqs where the number is greater than I specified at the command line (in this case: --n-procs 6). I don't think this is a bug in resource requests, though, as when I look at System Monitor, I see 6 processes (1 python + 5 ig-sw), so I assume what this means is that I have enough sequences that partis wants to split them into 9 pieces, even though only five will be run at once. If I'm right, may I suggest clarifying the message?

[psathyrella]

The short answer is that in some cases I'm taking --n-procs more as a suggestion than an order ;-)

Just going by memory here (I'll look through and see if I can make it more clear what's happening), but I think there's two cases where this is going to happen:

• in smith waterman, I (as of fairly recently) split up the queries by mutation rate when I pass to ig-sw, so that I can specify different match/mismatch scores for sequences with different shm rates. This was actually a pretty significant accuracy optimization. But this means that the ig-sw caller runs all those procs through the subprocess infrastructure at once (although as you say they aren't necessarily acutally concurrently running). The case that's obvious to me is if you ask for 1 proc, and it decides it needs three categories of match/mismatch score, it'll run three procs. But I don't remember the details and it could explain your case as well.

• when partitioning, the first step after sw is to precache hmm-derived naive sequences for each input sequence, and the function that decides how many procs to use for this doesn't apply --n-procs as a strict maximum. Now I for sure don't remember the details of why every one of that cavalcade of if/else clauses is there, but the basic reason it doesn't just take --n-procs is that it assumes that --n-procs is geared toward the number of initial procs to use for partitioning (which will then gradually be reduced in each partitioning step), which has a different set of requirements than precaching the naive sequences, which is just simple linear scaling. The linear scaling means that it has a more simple incentive to just maximize the number of processes, whereas for the actual partitioning, the more procs you start with, the more partitioning steps you have to run through, so there's some incentive to have fewer procs.

[psathyrella]

I added a note about this to the help for --n-procs. If you see it hammer the machine with an absurd number of processes, that's a bug, but as long as it's like 9 vs 6, then that's expected behavior.