search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

can't pickle _asyncio.Task objects #147

Open Mradr opened 6 years ago

Mradr commented 6 years ago 
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

Is there a way to tell where this error came from and is there a workaround to make this still work?

nymphias commented 6 years ago

Hi all,

I have a similar issue where the code fails at the same line with pickle error:

~/.local/lib/python3.6/site-packages/dill/_dill.py in save_module_dict(pickler, obj) 891 # we only care about session the first pass thru 892 pickler._session = False --> 893 StockPickler.save_dict(pickler, obj) 894 log.info("# D2") 895 return

/opt/conda/lib/python3.6/pickle.py in save_dict(self, obj) 819 820 self.memoize(obj) --> 821 self._batch_setitems(obj.items()) 822 823 dispatch[dict] = save_dict

/opt/conda/lib/python3.6/pickle.py in _batch_setitems(self, items) 845 for k, v in tmp: 846 save(k) --> 847 save(v) 848 write(SETITEMS) 849 elif n:

/opt/conda/lib/python3.6/pickle.py in save(self, obj, save_persistent_id) 494 reduce = getattr(obj, "__reduce_ex", None) 495 if reduce is not None: --> 496 rv = reduce(self.proto) 497 else: 498 reduce = getattr(obj, "reduce__", None)

TypeError: can't pickle dict_keys objects

I am not clear on which dict_keys object is being pickled here. I am using pathos.multiprocessing ProcessingPool function on a custom-class method. Thks!

mmckerns commented 5 years ago

@nymphias, @Mradr: It helps greatly if you can post a minimal, self-contained bit of code that produces your error. That way, it's easier for others to diagnose. The traceback is helpful, but a minimal, self-contained bit of code that produces your error is better.

In the abstract, what I can say is that it looks like you are running into items that can't be pickled. A ProcessingPool pickles the objects when it sends them to another process. You can try: (1) doing dill.check and dill.copy on one of your objects, to see if it fails serialization, (2) turn on dill.detect.trace to have it print out the serialization trace, (3) try one of the other pools, like ThreadPool (doesn't rely as heavily on serialization), or ParallelPool (extracts source code as opposed to serializing). Reporting what happens with any of the above is also good. There's a fourth option, which is to modify the object you are trying to serialize... but I can't give you specific advice there as I don't have an example of what is triggering the error you are seeing.

mmckerns commented 5 years ago

Also... sorry for the slow response here. Not sure why this wasn't responded to earlier.