Open kevinkhu opened 5 months ago
I was able to get a good speedup on Example 2 by running on full logical cores (8) instead of full physical cores (4). The first run of analysis.test_hypothesis_grid()
took about 9 minutes to run with 8 cores, but over 10 minutes to run with 4 cores.
The device was the MacBook Pro.
In my limited testing, instances that use multiprocessing should be limited to the number of physical CPU cores rather than number of threads. You can check how many physical cores you have with:
Set
logical = True
to count total threads. Can others test this on their machines? I ran Example 1 withprocesses=8
(physical cores in my laptop) andprocesses=16
(total threads) and 8 processes was faster by about 5 seconds. I am currently running my own test_hypothesis_grid code and by changing it to physical cores it should finish in <12 hours as opposed to several days using all threads. There might be a way to optimize using threads, but I am not sure how at this time.