search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

would you please provide an never stuck pool function? #229

Open showkeyjar opened 2 years ago

showkeyjar commented 2 years ago

Hi, I use pathos to solve my data process work,

it usually stuck or blocked at file w/r, network, sql w/r ...

it's too many exceptions to cache, and those exceptions makes me crazy.

so I wonder, if your guys can design a module or function that can not be stuck anytime?

it maybe very useful if this module be published.

hope your answer, best wishes!

mmckerns commented 2 years ago

What do you mean, never stuck? Basically, the map caller spawns jobs to the workers... and essentially listens/waits for a reply. Are you thinking that you would like a timed map that gives up if results don't come back in enough time, or what?

showkeyjar commented 2 years ago

@mmckerns yes,

I think we don't care a little bad data when we process batch works for big data, so there is no need to wait for all workers done, the bad or error data could be dropped.