Please consider the following example, where a subprocess tries to allocate more RAM than is available on the system. This snippet will hang/never finish/not raise an exception.
from pathos.multiprocessing import ProcessPool
from random import random
def outofmemory(*args):
mem = []
while True:
mem.append([random() for _ in range(10000)])
for nope in ProcessPool(2).imap(outofmemory, [1, 2]):
pass
In kubernetes clusters (also local dockerd if you set a container memory limit), subprocesses in a Pod will be SIGKILL'ed at any time and without warning to prevent PID 1 of the Pod going Out of Memory based on its resources.limits.memory spec (ref).
Can iterating over the imap raise an exception when one of the subprocesses dies unexpectedly (so not due to maxtasksperchild)?
Hi 👋
Please consider the following example, where a subprocess tries to allocate more RAM than is available on the system. This snippet will hang/never finish/not raise an exception.
In kubernetes clusters (also local dockerd if you set a container memory limit), subprocesses in a Pod will be SIGKILL'ed at any time and without warning to prevent PID 1 of the Pod going Out of Memory based on its
resources.limits.memoryspec (ref).Can iterating over the imap raise an exception when one of the subprocesses dies unexpectedly (so not due to
maxtasksperchild)?Originally posted by @thoughtfuldata in https://github.com/ddelange/mapply/issues/43