Add a mechanism to automatically retry failed tasks

It would be useful to have a way to retry failed tasks automatically as soon as they fail - and not have to manually resubmit them at the end.

I've implemented such a mechanism here, which I'd be happy to open a PR to donate it to Lithops.

The implementation uses wrappers around ResponseFuture and FunctionExecutor, so you use it like this:

function_executor = LocalhostExecutor()
with RetryingFunctionExecutor(function_executor) as executor:
    futures = executor.map(
        function,
        input,
        timeout=timeout,
        retries=retries,
    )
    done, pending = executor.wait(futures)
    ...

RetryingFuture has to have a reference to the function and the input so it can submit a new task if the original fails - so it makes sense to use a difference class to ResponseFuture. However, it might be possible to extend FunctionExecutor to take a retries argument, and only use the more heavyweight RetryingFuture if it is set. Or perhaps it's OK to add fields to ResponseFuture and only set them if retries are enabled?

lithops-cloud / lithops

Add a mechanism to automatically retry failed tasks #1289