search

popgenmethods / pyrho

Fast inference of fine-scale recombination rates based on fused-LASSO
MIT License
43 stars 4 forks source link

Hyperparam not using threads #14

Open lczech opened 4 years ago

lczech commented 4 years ago

When running the hyperparam command

pyrho hyperparam -n 20 --mu 1.25e-08 --blockpenalty 50,100 --windowsize 25,50 \
--num_sims 3 --tablefile out/ACB.hdf --smcpp_file example/ACB_pop_sizes.csv \
--numthreads 6 --outfile out/ACB_hyper.txt --logfile logs/hyperparam/ACB.log

with the exemplary dataset, which runs for ~90sec on my laptop, it seems as if only a single thread is being used the whole time:

Screenshot from 2020-06-18 14-32-45

The make_table command on the other hand uses all threads, so it probably is not an issue with my general setup (although of course I might have missed something).

NB: As of now, I cannot judge yet whether threads are used for the optimize command, as this is too quick with the exemplary dataset. If this also does not use threads, I'll report here as well.

jeffspence commented 4 years ago

There's some batching to prevent multiprocessing overhead. It's pretty aggressive for hyperparam. If you make num_sims > 50, then more threads will be used. In general it should use up to num_sims / 50 threads if you set numthreads high enough. I'll leave this open until I make the batching better.