search

soravux / scoop

SCOOP (Scalable COncurrent Operations in Python)
https://github.com/soravux/scoop
GNU Lesser General Public License v3.0
625 stars 88 forks source link

how to run a task on each worker? #56

Open joernhees opened 7 years ago

joernhees commented 7 years ago

is there an easy way to run a function on each worker?

I have a scenario where i'd like to trigger writing stats on program termination on each worker. Each of the workers has a local cache etc. and syncing the stats during computation would cause a lot of stupid overhead.

soravux commented 7 years ago

There is currently no way in the API to do this explicitly. API proposals for this are welcome for discussion.

To solve your problem, you could either add a callback to your futures or, at the end of your futures, perform a stats write if the last time it was performed is longer than a desired amount of time.