Can't pickle local object 'starargs.<locals>.<lambda>'

[phmarch]

So I've been trying to get pathos to work on my windows environment and I keep getting the same error for the simple examples I found in the doc:

from pathos.multiprocessing import ProcessingPool
class Bar:
    def foo(self, name):
        return len(str(name))
    def boo(self, things):
        for thing in things:
            self.sum += self.foo(thing)
        return self.sum
    sum = 0

b = Bar()
results = ProcessingPool().map(b.boo, [[12,3,456],[8,9,10],['a','b','cde']])

I get this error:

AttributeError                            Traceback (most recent call last)
<ipython-input-1-7461caccb424> in <module>
     10 
     11 b = Bar()
---> 12 results = ProcessingPool().map(b.boo, [[12,3,456],[8,9,10],['a','b','cde']])
     13 results

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pathos\multiprocessing.py in map(self, f, *args, **kwds)
    135         AbstractWorkerPool._AbstractWorkerPool__map(self, f, *args, **kwds)
    136         _pool = self._serve()
--> 137         return _pool.map(star(f), zip(*args)) # chunksize
    138     map.__doc__ = AbstractWorkerPool.map.__doc__
    139     def imap(self, f, *args, **kwds):

~\AppData\Local\Continuum\anaconda3\lib\multiprocessing\pool.py in map(self, func, iterable, chunksize)
    266         in a list that is returned.
    267         '''
--> 268         return self._map_async(func, iterable, mapstar, chunksize).get()
    269 
    270     def starmap(self, func, iterable, chunksize=None):

~\AppData\Local\Continuum\anaconda3\lib\multiprocessing\pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658 
    659     def _set(self, i, obj):

~\AppData\Local\Continuum\anaconda3\lib\multiprocessing\pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
    429                         break
    430                     try:
--> 431                         put(task)
    432                     except Exception as e:
    433                         job, idx = task[:2]

~\AppData\Local\Continuum\anaconda3\lib\multiprocessing\connection.py in send(self, obj)
    204         self._check_closed()
    205         self._check_writable()
--> 206         self._send_bytes(_ForkingPickler.dumps(obj))
    207 
    208     def recv_bytes(self, maxlength=None):

~\AppData\Local\Continuum\anaconda3\lib\multiprocessing\reduction.py in dumps(cls, obj, protocol)
     49     def dumps(cls, obj, protocol=None):
     50         buf = io.BytesIO()
---> 51         cls(buf, protocol).dump(obj)
     52         return buf.getbuffer()
     53 

AttributeError: Can't pickle local object 'starargs.<locals>.<lambda>'

My environment:

• Windows 7
• Python 3.7
• pathos 0.2.3

Note that I installed pathos with pip install pathos and it worked fine. The dependencies are:

pathos (0.2.3)
ppft>=1.6.4.9 (1.6.4.9)
multiprocess>=0.70.7 (0.70.7)
pox>=0.2.5 (0.2.5)
dill>=0.2.9 (0.2.9)
six>=1.7.3 (1.12.0)

[mmckerns]

I'm not able to reproduce your error with a similar environment. Do you have a C compiler installed? Without a C compiler, pathos will install fine, but may throw a warning that _multiprocess is not installed -- that is unless you used the multiprocess wheel upon install, which I believe pip does for you.

The error line in the traceback raise self._value is indicative of not having the _multiprocess module installed (which requires the wheel or a C compiler).

Try: import _multiprocess

If that fails, you have an incomplete install of multiprocess, and likely need a C compiler or to use the wheel of multiprocess.

[phmarch]

I just tried to reproduce the error and I think I understand why I got it. If I run the code I pasted by itself, it works great but if I import multiprocessing before, I get the error again.

Since I am using jupyter notebook, a single import multiprocess will be kept in memory and will prevent me from using pathos until the notebook is reset, which is why I overlooked this detail while looking for the origin of the problem.

Somehow import _multiprocessing works but I still get the error. I just cannot use multiprocessing and pathos in the same notebook.

I tried in a standalone python script and both of them seem to work without interfering. I guess it's a weird interaction with jupyter notebook.

Thank you

[mmckerns]

Jupyter notebooks do strange things... basically, each individual cell is kind of like an independent file... but at the end, after it executes, it dumps it's globals into the notebook's globals.

pathos and multiprocess shouldn't conflict... pathos uses multiprocess. Did you check import _multiprocess or import _multiprocessing? They are different. I meant the former. The latter is from python STL.

Note the use of both multiprocess and pathos in the same notebook here: https://github.com/mmckerns/tuthpc/blob/master/multiprocessing.ipynb

[phmarch]

Oooh I found the issue. I had a multiprocess.py file in the same folder (that I randomly named to test out pathos) and the import multiprocess must have been importing this one instead of the library.

import _multiprocess isn't throwing any errors so I'm sure this was the problem! Both multiprocessing and pathos are now working in the same notebook. Thank you

[Abhishek-kumar-k]

AttributeError: Can't pickle local object 'run..' how to solve?

[Abhishek-kumar-k]

---

AttributeError Traceback (most recent call last)

in 288 "patience": 5 289 }) --> 290 run(args) in run(args) 212 t_start = time.time() 213 --> 214 train_loss, train_auc = train_model(mrnet, train_loader, epoch, num_epochs, optimizer, writer, current_lr, log_every) 215 val_loss, val_auc = evaluate_model(mrnet, validation_loader, epoch, num_epochs, writer, current_lr) 216 in train_model(model, train_loader, epoch, num_epochs, optimizer, writer, current_lr, log_every) 33 losses = [] 34 ---> 35 for i, (image, label, weight) in enumerate(train_loader): 36 optimizer.zero_grad() 37 ~\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __iter__(self) 277 return _SingleProcessDataLoaderIter(self) 278 else: --> 279 return _MultiProcessingDataLoaderIter(self) 280 281 @property ~\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __init__(self, loader) 717 # before it starts, and __del__ tries to join but will get: 718 # AssertionError: can only join a started process. --> 719 w.start() 720 self._index_queues.append(index_queue) 721 self._workers.append(w) ~\Anaconda3\lib\multiprocessing\process.py in start(self) 110 'daemonic processes are not allowed to have children' 111 _cleanup() --> 112 self._popen = self._Popen(self) 113 self._sentinel = self._popen.sentinel 114 # Avoid a refcycle if the target function holds an indirect ~\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj) 221 @staticmethod 222 def _Popen(process_obj): --> 223 return _default_context.get_context().Process._Popen(process_obj) 224 225 class DefaultContext(BaseContext): ~\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj) 320 def _Popen(process_obj): 321 from .popen_spawn_win32 import Popen --> 322 return Popen(process_obj) 323 324 class SpawnContext(BaseContext): ~\Anaconda3\lib\multiprocessing\popen_spawn_win32.py in __init__(self, process_obj) 87 try: 88 reduction.dump(prep_data, to_child) ---> 89 reduction.dump(process_obj, to_child) 90 finally: 91 set_spawning_popen(None) ~\Anaconda3\lib\multiprocessing\reduction.py in dump(obj, file, protocol) 58 def dump(obj, file, protocol=None): 59 '''Replacement for pickle.dump() using ForkingPickler.''' ---> 60 ForkingPickler(file, protocol).dump(obj) 61 62 # AttributeError: Can't pickle local object 'run..'