search

Circuitscape / Circuitscape.jl

Algorithms from circuit theory to predict connectivity in heterogeneous landscapes
https://circuitscape.org
MIT License
128 stars 35 forks source link

Circuitscape not efficiently utilizing all cores during parallel processing? #165

Closed vlandau closed 4 years ago

vlandau commented 5 years ago

@ranjanan When running Circuitscape in parallel, it appears based on the info that is printed to the console that the last several tasks are solved in serial rather that parallel. For example, the first 12 pairs may be solved 4 at a time, as expected, but the last 8 pairs are solved only two or one at a time.

It's not a big deal when solve times are only a few seconds, but in my case solves take about 15 minutes each (I'm using Google Compute Engine which charges per second of VM use, so the time (and money) savings gained by fully utilizing all available cores would be significant).

Thanks for any insight!

vlandau commented 5 years ago

Output for the most extreme example is shown here. In this case, I've set Circuitscape to use 4 parallel processes and given it only 4 node pairs to solve. Instead of sending one to each process and solving them simultaneously, they are solved in serial. I'd be happy to email the inputs for this example as well. This is all run within a Docker container (Ubuntu 16.04 and Julia 1.0.2).

root@fa4059c5dd53:/home# julia -e 'using Circuitscape; compute("/home/usr/input/1km.ini")' [ Info: 2018-12-11T02:35:56.085 : Precision used: Double [ Info: 2018-12-11T02:35:56.476 : Starting up Circuitscape to use 4 processes in parallel [ Info: 2018-12-11T02:36:10.151 : Reading maps [ Info: 2018-12-11T02:36:11.916 : Resistance/Conductance map has 410997 nodes [ Info: 2018-12-11T02:36:22.602 : Solver used: AMG accelerated by CG [ Info: 2018-12-11T02:36:22.602 : Graph has 410997 nodes, 5 focal points and 1 connected components [ Info: 2018-12-11T02:36:22.604 : Total number of pair solves = 10 [ Info: 2018-12-11T02:36:24.774 : Time taken to construct preconditioner = 2.0054636 seconds [ Info: 2018-12-11T02:36:25.111 : Time taken to construct local nodemap = 0.3284378 seconds [ Info: 2018-12-11T02:36:55.068 : Scheduling pair 8 of 10 to be solved [ Info: 2018-12-11T02:36:56.544 : Scheduling pair 9 of 10 to be solved [ Info: 2018-12-11T02:36:55.068 : Scheduling pair 5 of 10 to be solved [ Info: 2018-12-11T02:36:55.068 : Scheduling pair 10 of 10 to be solved [ Info: 2018-12-11T02:36:55.068 : Scheduling pair 1 of 10 to be solved [ Info: 2018-12-11T02:36:56.551 : Scheduling pair 2 of 10 to be solved [ Info: 2018-12-11T02:36:56.552 : Scheduling pair 3 of 10 to be solved [ Info: 2018-12-11T02:36:56.552 : Scheduling pair 4 of 10 to be solved [ Info: 2018-12-11T02:36:56.559 : Scheduling pair 6 of 10 to be solved [ Info: 2018-12-11T02:36:56.559 : Scheduling pair 7 of 10 to be solved [ Info: 2018-12-11T02:36:56.611 : Solving pair 1 of 10 [ Info: 2018-12-11T02:36:59.999 : Time taken to solve linear system = 3.3659902 seconds [ Info: 2018-12-11T02:37:30.901 : Time taken to write current maps = 30.4863212 seconds [ Info: 2018-12-11T02:37:31.052 : Solving pair 2 of 10 [ Info: 2018-12-11T02:37:33.175 : Time taken to solve linear system = 2.1225458 seconds [ Info: 2018-12-11T02:37:34.72 : Time taken to write current maps = 1.5445566 seconds [ Info: 2018-12-11T02:37:34.722 : Solving pair 3 of 10 [ Info: 2018-12-11T02:37:36.782 : Time taken to solve linear system = 2.0599558 seconds [ Info: 2018-12-11T02:37:38.293 : Time taken to write current maps = 1.5110374 seconds [ Info: 2018-12-11T02:37:38.294 : Solving pair 4 of 10 [ Info: 2018-12-11T02:37:40.452 : Time taken to solve linear system = 2.1574078 seconds [ Info: 2018-12-11T02:37:42.073 : Time taken to write current maps = 1.5789583 seconds [ Info: 2018-12-11T02:42:22.13 : Time taken to complete job = 374.8844891

ranjanan commented 5 years ago

@vlandau thanks for pointing this out. Can you send me an email with the inputs so I can reproduce?

vlandau commented 5 years ago

@ranjanan Thanks for the speedy response. Just emailed you.

ViralBShah commented 5 years ago

Is this still an issue? Might be fixed.

vlandau commented 5 years ago

@ViralBShah, not sure. @ranjanan?

vlandau commented 4 years ago

Not sure if this is fixed yet, but it should automatically be resolved once multi-threading is implemented (I would think).

k-barnett commented 4 years ago

Hi all, just wanted to follow up on this to see if Circuitscape is in fact using all processors declared in the .ini files. From my end, I don't see any change in performance between 8 and 24 processors when performing a pairwise model run. Running on Windows, Julia 1.4.1.

vlandau commented 4 years ago

@k-barnett please feel free to open a new issue and we'll try to get it sorted for you!

vlandau commented 4 years ago

I'm closing this as the parallel processing framework will change with multithreading. Can open a new issue if a similar issue arises then.