Implement `error_callback` for `map`

[ma-sadeghi]

It'd be great if map accepts error_callback as argument.

[sybrenjansen]

Can you explain what the use case would be for this?

If something fails, then the main process will raise and you could simply catch that and process it as desired. Or would you want it to not raise and only call the callback function? The map call would then return None or something? It would help if I understood the use case

[ma-sadeghi]

Thanks for your reply. I think I was trying to use map in an unusual way, here's my use case:

I'm using map to distribute an embarrassingly parallel workload that occasionally gets stuck (eg takes 10x longer than average). I'd like to discard those long running processes. I was trying to achieve that with mpire's timeout argument, but that raises an Exception, terminating the master process. I was trying to work around that by handling the TimeoutException using callback, which is not implemented atm.

Please let me know if it wasn't clear, I can share a MWE when I'm at my desk.

Thanks!

[sybrenjansen]

One workaround for this now is to use apply_async. But then some of the functionality will of course not be available (e.g., progress bar). Is that why you want to use map instead?

I could add an error callback option, but that would then automatically imply that nothing will be raised whenever an error is raised from one of the tasks. Be it a runtime error or a task timeout. All errors would be forwarded to the error callback. Perhaps I could add the feature that based on the return value of the callback, the workers continue with the left over tasks or stop completely.

Before I start working on this though, I need to work on something else first, which should make the work required for this a bit easier.