search

danielapai / bioverse

A simulation framework to assess the statistical power of future biosignature surveys
MIT License
7 stars 5 forks source link

Use only physical cores in multiprocessing #50

Open kevinkhu opened 5 months ago

kevinkhu commented 5 months ago

In my limited testing, instances that use multiprocessing should be limited to the number of physical CPU cores rather than number of threads. You can check how many physical cores you have with: 

import psutil 
psutil.cpu_count(logical = False)

Set logical = True to count total threads. Can others test this on their machines? I ran Example 1 with processes=8 (physical cores in my laptop) and processes=16 (total threads) and 8 processes was faster by about 5 seconds. I am currently running my own test_hypothesis_grid code and by changing it to physical cores it should finish in <12 hours as opposed to several days using all threads. There might be a way to optimize using threads, but I am not sure how at this time.

astral-carrier commented 3 months ago

I was able to get a good speedup on Example 2 by running on full logical cores (8) instead of full physical cores (4). The first run of analysis.test_hypothesis_grid() took about 9 minutes to run with 8 cores, but over 10 minutes to run with 4 cores.

The device was the MacBook Pro.