Open berland opened 5 years ago
concurrent.futures
should be used for this. Needs a backport for Python 2.7.
Would it be an idea to not support Python 2.7 (just leave the old code in place when running Python 2.7) and only build this for Python3?
A suggestion could be to allow for more processing in a realization to happen at time of object initialization. It might be possible to pass a dict with names of realization function call as keys, and with (list of) function arguments as dict values, which can be passed to __init__
, and that would enable calling each necessary load_*
function concurrently. __init__
in a realization would use a "batch_processor
" in the realization object that can also serve as a general wrapper for later concurrent operations, and this function should return the realization object when finished, to be compatible with concurrent.future.
Batch processor in #78
Operations over an ensembles are trivially parallelizable.
We should utilize Python multiprocessing for this.
multiprocessing is what should be used, as multithreading will suffer from GIL.
This is probably trivial for
ensemble.get_smry()
, but not so trivial forensemble.from_smry()
, as we need to populate each realization object with smry data in the parent process' memory space.Maybe
ensemble.from_smry()
should callrealization.get_smry()
with multiprocessing, and then the ensemble object (holding the master process) populates each realizationsself.data['unsmry-<something>']
.We must ensure CTRL-C works, which is trickier with Multiprocessing.
See this: https://stackoverflow.com/questions/11312525/catch-ctrlc-sigint-and-exit-multiprocesses-gracefully-in-python
When this is in place, we should also be able to skip issues when libecl is core-dumping due to a difficult UNSMRY-file.
Right now, your Python session will die if libecl crashes on rough data.