search

dereneaton / ipyrad

Interactive assembly and analysis of RAD-seq data sets
http://ipyrad.readthedocs.io
GNU General Public License v3.0
72 stars 40 forks source link

0.9.0 - detect_cpus() is wonky #332

Closed isaacovercast closed 5 years ago

isaacovercast commented 5 years ago

Clean install of 0.9.0 branch running step 1 with SE RAD sim data. This is on a 40 core box.

(py36) casey@cattywampus:~/tmp/ipyrad$ ipyrad -p params-watdo.txt -s 1 -f

 -------------------------------------------------------------
  ipyrad [v.0.9.0-dev]
  Interactive assembly and analysis of RAD-seq data
 ------------------------------------------------------------- 
Parallelization: cattywampus: 8 cores

  Step 1: Demultiplexing fastq data to Samples                                                                                     
  [####################] 100% 0:00:09 | sorting reads                                                                              
  [####################] 100% 0:00:03 | writing/compressing                                                                        
(py36) casey@cattywampus:~/tmp/ipyrad$ ipyrad -p params-watdo.txt -s 1 -f                                                          

 -------------------------------------------------------------                                                                     
  ipyrad [v.0.9.0-dev]                                                                                                             
  Interactive assembly and analysis of RAD-seq data                                                                                
 -------------------------------------------------------------                                                                     
Parallelization: cattywampus: 12 cores                                                                                             

  Step 1: Demultiplexing fastq data to Samples                                                                                     
  [####################] 100% 0:00:09 | sorting reads                                                                              
  [####################] 100% 0:00:03 | writing/compressing
isaacovercast commented 5 years ago

Correction, detect_cpus() is fine, but the reporting on how many cpus are available is wonky. You can see the ipcluster instance does have 40 cores:

casey    23446     1  5 23:29 ?        00:00:07 /home/casey/miniconda2/envs/py36/bin/python /home/casey/miniconda2/envs/py36/bin/ipcluster start --daemonize --cluster-id=ipyrad-cli-397 --engines=Local --profile=default --n=40
isaacovercast commented 5 years ago

This feels like a timing issue.

eaton-lab commented 5 years ago

Yeah, I shortened the wait timing when I was messing with those widget based progressbars, since you could have it just update the number of cores on there as they come active. But I don't see the widget bars being integrated into ipyrad so we should make sure it finds all the cores before it moves forward.

isaacovercast commented 5 years ago

Seems to work fine now.