mmckerns
commented
5 years ago @daeh: In running your initial code, I receive the same error as you do, so that's verified. You also make the comment about running this code (I believe):
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>>
>>> def model_handler(n):
... import matplotlib
... matplotlib.use('agg')
... import matplotlib.pyplot as plt
... figout = plt.figure(figsize=(1, 1))
... axes = figout.add_subplot(1,1,1)
... axes.scatter(range(1000), range(1000))
... figout.savefig('dummyfig{}.pdf'.format(n), format='pdf')
... return n
...
>>> dataset_serial = map(model_handler, range(20))
>>> datasets = Pool(20).map(model_handler, range(20))
>>> list(dataset_serial)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> As you can see, I don't receive an error in the above. Is that what you are reporting?
As to why there's an error in when the serial version runs first, compared to when the parallel version runs first... I expect that is due to lazy evaluation of the model_handler function. When a function is first executed, it produces new python objects, and potentially changes (1) what's pointed to by globals, as well as (2) what needs to be passed to the other processors to be run in parallel. So, I expect it's some issue with regard to that. What exactly is going on, I can't say yet, but I assume it's a serialization issue.
The traceback that you are seeing with raise self._value is indicative of a failure to exchange a serialized object between the two processes. This really isn't a pathos issue, but more likely either a multiprocess issue, or a dill issue, or a matplotlib issue... depending on what exactly is the problem.
lisette-espin
marcotama
I'm running into an issue trying to save matplotlib figures as PDF in parallel. The behavior is a bit odd and I'm not sure if this is a pathos issue, per se, but pathos is the only place I've encountered it. The minimal code below causes a few related errors in set_text and load_char
Will throw
Interestingly, if I replace the dataset_serial loop with
dataset_serial = map(model_handler, range(20)), the pathos parallel pool executes with no problem, but if I convert the serial map object to a list (print(list(dataset_serial))), the glyph exception in the parallel pool comes back. Any thoughts as to what's going on here?