look into the "loky" ProcessPoolExecutor to replace multiprocessing.Process in transformation execution

[sjdv1982]

Should be compatible with multiprocessing.Queue Could be used in other tasks (especially checksum calculation and serialization)

Would be a fix for #128

[sjdv1982]

After looking into loky, I arrived at the following conclusions:

• Loky is not robust enough. It is essential that a Seamless job can fail horribly without any impact on the rest of the system. However, if any worker in the loky scheduler receives e.g. a SIGTERM, this will shut down all running jobs. There is only one singleton loky scheduler.
• loky claims to do fork+exec, but it still does a lot of pickling. Arguments are pickled. Global state is pickled (if possible?). There is a "fork" start method, but it is 1) not recommended and 2) does not reproduce the global state at the time of job launch.
• It is not compatible with multiprocessing queues (they cannot be pickled). There seem to be no native IPC queue mechanism. On the other hand, results are accessible directly, so are Python exceptions (including SysExitError), and print-to-terminal works out of the box.

Therefore, loky can be a useful new (i.e. additional) backend to locally execute Seamless jobs. A job definitely needs to be annotated (e.g. {"meta": "simple"} as suitable for loky. In particular:

• No compiled code (too much risk of SIGXXX). Only code that raises Python exceptions, not sure about bash/R/etc.
• No modules.
• No re-entrance of any kind: no seamless.imperative, no preliminary. Progress bar might be copyable from joblib though.
• No big data, since lock cannot be prematurely released.
• Probably no debugging On the other hand, if it works, it should fix fork deadlocks and might be usable under Windows too.

[sjdv1982]

Now that delegation works fine, the risk of forking can be easily mitigated. loky is no longer expected to be needed.