search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

use id keyword to enable multiple cached pools #213

Closed mmckerns closed 3 years ago

mmckerns commented 3 years ago

Pools are cached in pathos. So, when a second pool is created of the same type, it actually is just a pointer to the cached pool. When the "new" pool has a different number of nodes (say, M are requested, where the existing has N), it still doesn't create a new pool, it converts the existing pool from using N nodes to M nodes. This behavior can run in to memory issues if you don't manage the processes well (close and join them, appropriately). Let's use the id keyword to have a default to the number of nodes, so by default, requests for pools with different nodes create new cached pools. This was my original intent, but I didn't get around to finishing the implementation.

mmckerns commented 3 years ago

Done in 691ada8d32391deae216fc6e76e55025aa622c31. The "old" behavior is still possible by giving a fixed id. Say, id='fixed' for all pools. If one doesn't give an id, then the pools will cache for each pool with a different number of nodes. If id is given something different (e.g. a random number) each time, then all pools will be cached as independent, even if the configuration is the same.