pickler error

[minhhg]

Dear Folks, I have following errors using pathos import pathos.multiprocessing as mp pool = mp.ProcessingPool(2) res = pool.map(do_optimize1b, x0s) on ubuntu 16.04, 64bits, anaconda,

pathos 0.2a1.dev0 pp-1.5.7 pathos processing-0.52 pathos pyre-0.8.2.0 pathos

Is there any way to debug this as the code is a bit complicated to attach it here? Many thanks, Minh

Exception in thread Thread-1: Traceback (most recent call last): File "/home/minh/miniconda/envs/banks/lib/python2.7/threading.py", line 810, in bootstrap_inner self.run() File "/home/minh/miniconda/envs/banks/lib/python2.7/threading.py", line 763, in run self.target(*self.args, **self.__kwargs) File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/processing-0.52_pathos-py2.7-linux-x8664.egg/processing/pool.py", line 207, in handleTasks put(task) File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 192, in dumps dump(obj, file, protocol, byref, fmode)#, strictio) File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 182, in dump pik.dump(obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 224, in dump self.save(obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 626, in save_function obj.dict__), obj=obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 401, in save_reduce save(args) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 826, in save_cell pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 401, in save_reduce save(args) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 626, in save_function obj.dict), obj=obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 401, in save_reduce save(args) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 826, in save_cell pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj) ValueError: Cell is empty

[mmckerns]

The best way to get help is to provide a minimal version of the code that still reproduces the error.

However, there are a few things you can do to try to solve it yourself (or provide more information here). If you change: multiprocessing.ProcessingPool(2) to pools.ThreadPool(1) does it work? If so, it's a pickling problem. If it's not a pickling problem, then you can replace the pathos map with a standard python map and debug normally.

If the ThreadPool works, then switch to pools.ProcessPool(1) and add the line import dill; dill.detect.trace(True). This will spit out a bunch of trace information about what it being serialized, and you can see what the code is choking on. That might give you some indication of how to change your code to make it pass. From the trace, I can see a lot of tuple objects being dumped, but it looks like the thing chokes on a cell object... which is typically something that is passed to a helper function that is being pickled (which I can also see in your trace). I'd need to see the full trace (as I suggested above) to tell you more. The typical reason for seeing the error you posted is that the function to be pickled has free variables that are referenced before assignment in enclosing scope. I'd need to see code to tell you more.

If you don't want to post code, or you solve it, please let me know by closing the ticket.

[minhhg]

Hi Mike, Thanks for your reply. I change mp.ProcessingPool(2) to mp.ThreadPool(1). The problem still remains. If I change to standard multiprocessing map. I got the following import multiprocessing as mp pool = mp.Pool(2)

Traceback (most recent call last): File "ev2.py", line 16, in res = optim1(x0) File "/home/minh/python/eventlet/ev1.py", line 53, in optim1 res = do_optimize(x0,fun, nps=2, nstart_points=10, max_obj=False, options=xoptions) File "/home/minh/dev/master/python/ev/model/estimators/interest_rates/gl_gp_com.py", line 696, in do_optimize res = pool.map(do_optimize1c, x0s) File "/home/minh/miniconda/envs/banks/lib/python2.7/multiprocessing/pool.py", line 251, in map return self.map_async(func, iterable, chunksize).get() File "/home/minh/miniconda/envs/banks/lib/python2.7/multiprocessing/pool.py", line 567, in get raise self._value cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup builtin.function failed

[minhhg]

Here are the output if I add import dill; dill.detect.trace(True) mp.ProcessingPool(1)

============================================= (banks) minh:eventlet$python ev2.py /home/minh/miniconda/envs/banks/lib/python2.7/site-packages/sklearn/lda.py:4: DeprecationWarning: lda.LDA has been moved to discriminant_analysis.LinearDiscriminantAnalysis in 0.17 and will be removed in 0.19 "in 0.17 and will be removed in 0.19", DeprecationWarning) /home/minh/miniconda/envs/banks/lib/python2.7/site-packages/sklearn/qda.py:4: DeprecationWarning: qda.QDA has been moved to discriminant_analysis.QuadraticDiscriminantAnalysis in 0.17 and will be removed in 0.19. "in 0.17 and will be removed in 0.19.", DeprecationWarning) ('lb=', array([-1.4226973834243688, -3.6522752635392801, -0.5916678091519202, -8.0080464313820876, -0.6512982708321773, -6.4639854203026035, -3.5111542494302128, -2.0202811170209714, -0.673176631166826 , -5.6270790433573961])) ('ub=', array([ 1.7388523575186732, 4.4638919887702313, 0.7231495445190136, 9.7876123050225523, 0.7960312199059945, 7.9004266248142931, 4.2914107493035933, 2.4692324763589655, 0.8227714380927874, 6.8775410529923739])) ('scale=', array([ 0.3975895962511495, 0.6370308961231594, 0.2563998199756519, 0.9432830629351043, 0.2690101755267049, 0.8474789686215493, 0.6246024735275152, 0.473788644512505 , 0.2734911396425498, 0.7907155018193892])) ('#nproess=', 2, ' #points=', 10) F2: <function mapstar at 0x7fb5fa506a28> F2: <function mapstar at 0x7fb5fa506a28> F1: <function at 0x7fb5ec4129b0> F1: <function at 0x7fb5ec4129b0> F2: <function _create_function at 0x7fb5fa55fe60> F2: <function _create_function at 0x7fb5fa55fe60> Co: <code object at 0x7fb5fa9a4030, file "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/pathos-0.2a1.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 13> Co: <code object at 0x7fb5fa9a4030, file "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/pathos-0.2a1.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 13> F2: <function _unmarshal at 0x7fb5fa55fcf8> F2: <function _unmarshal at 0x7fb5fa55fcf8> D4: <dict object at 0x7fb5fa538168> D4: <dict object at 0x7fb5fa538168> Ce: <cell at 0x7fb5ec400c20: function object at 0x7fb5ec4035f0> Ce: <cell at 0x7fb5ec400c20: function object at 0x7fb5ec4035f0> F2: <function _create_cell at 0x7fb5fa5631b8> F2: <function _create_cell at 0x7fb5fa5631b8> F1: <function do_optimize1c at 0x7fb5ec4035f0> F1: <function do_optimize1c at 0x7fb5ec4035f0> Co: <code object do_optimize1c at 0x7fb601841730, file "/home/minh/dev/master/python/ev/model/estimators/interest_rates/gl_gp_com.py", line 662> Co: <code object do_optimize1c at 0x7fb601841730, file "/home/minh/dev/master/python/ev/model/estimators/interest_rates/gl_gp_com.py", line 662> D4: <dict object at 0x7fb60182e398> D4: <dict object at 0x7fb60182e398> Ce: <cell at 0x7fb5ec400408: numpy.ndarray object at 0x7fb5ec3ff580> Ce: <cell at 0x7fb5ec400408: numpy.ndarray object at 0x7fb5ec3ff580> B2: B2: T4: <type 'numpy.ndarray'> T4: <type 'numpy.ndarray'> T4: <type 'numpy.dtype'> T4: <type 'numpy.dtype'> Ce: <cell at 0x7fb5ec400440: bool object at 0x7fb60a70a8b0> Ce: <cell at 0x7fb5ec400440: bool object at 0x7fb60a70a8b0> Ce: <cell at 0x7fb5ec4004b0: function object at 0x7fb5ec403320> Ce: <cell at 0x7fb5ec4004b0: function object at 0x7fb5ec403320> F1: <function myobj at 0x7fb5ec403320> F1: <function myobj at 0x7fb5ec403320> Co: <code object myobj at 0x7fb6018412b0, file "/home/minh/dev/master/python/ev/model/estimators/interest_rates/gl_gp_com.py", line 484> Co: <code object myobj at 0x7fb6018412b0, file "/home/minh/dev/master/python/ev/model/estimators/interest_rates/gl_gp_com.py", line 484> D4: <dict object at 0x7fb60182e398> D4: <dict object at 0x7fb60182e398> Ce: <cell at 0x7fb5ec400360: function object at 0x7fb5ec4032a8> Ce: <cell at 0x7fb5ec400360: function object at 0x7fb5ec4032a8> F1: <function fun at 0x7fb5ec4032a8> F1: <function fun at 0x7fb5ec4032a8> Co: <code object fun at 0x7fb60a836830, file "/home/minh/python/eventlet/ev1.py", line 41> Co: <code object fun at 0x7fb60a836830, file "/home/minh/python/eventlet/ev1.py", line 41> D4: <dict object at 0x7fb60333e6e0> D4: <dict object at 0x7fb60333e6e0> D2: <dict object at 0x7fb5ec4186e0> D2: <dict object at 0x7fb5ec4186e0> Ce: <cell at 0x7fb5ec4003d0: bool object at 0x7fb60a70a8b0> Ce: <cell at 0x7fb5ec4003d0: bool object at 0x7fb60a70a8b0> Ce: <cell at 0x7fb5ec4004e8: numpy.ndarray object at 0x7fb5ec3ff670> Ce: <cell at 0x7fb5ec4004e8: numpy.ndarray object at 0x7fb5ec3ff670> D2: <dict object at 0x7fb5ec4185c8> D2: <dict object at 0x7fb5ec4185c8> Ce: <cell at 0x7fb5ec400520: numpy.ndarray object at 0x7fb5ec3ff710> Ce: <cell at 0x7fb5ec400520: numpy.ndarray object at 0x7fb5ec3ff710> Ce: <cell at 0x7fb5ec400558: function object at 0x7fb5ec403500> Ce: <cell at 0x7fb5ec400558: function object at 0x7fb5ec403500> F1: <function x_2_solution at 0x7fb5ec403500> F1: <function x_2_solution at 0x7fb5ec403500> Co: <code object x_2_solution at 0x7fb6018414b0, file "/home/minh/dev/master/python/ev/model/estimators/interest_rates/gl_gp_com.py", line 583> Co: <code object x_2_solution at 0x7fb6018414b0, file "/home/minh/dev/master/python/ev/model/estimators/interest_rates/gl_gp_com.py", line 583> D4: <dict object at 0x7fb60182e398> D4: <dict object at 0x7fb60182e398> Ce: <cell at 0x7fb5ec4001d8: empty> Ce: <cell at 0x7fb5ec4001d8: empty> Exception in thread Thread-1: Traceback (most recent call last): File "/home/minh/miniconda/envs/banks/lib/python2.7/threading.py", line 810, in bootstrap_inner self.run() File "/home/minh/miniconda/envs/banks/lib/python2.7/threading.py", line 763, in run self.target(*self.args, **self.__kwargs) File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/processing-0.52_pathos-py2.7-linux-x86_64.egg/processing/pool.py", line 207, in _handleTasks put(task) File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 192, in dumps dump(obj, file, protocol, byref, fmode)#, strictio) File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 182, in dump pik.dump(obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 224, in dump self.save(obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 626, in save_function obj.dict), obj=obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 401, in save_reduce save(args) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 826, in save_cell pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 401, in save_reduce save(args) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 626, in save_function obj.dict), obj=obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 401, in save_reduce save(args) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 826, in save_cell pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 401, in save_reduce save(args) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 548, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 626, in save_function obj.dict__), obj=obj) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 401, in save_reduce save(args) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 562, in save_tuple save(element) File "/home/minh/miniconda/envs/banks/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/minh/miniconda/envs/banks/lib/python2.7/site-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 826, in save_cell pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj) ValueError: Cell is empty

[minhhg]

I try to provide a minimal version of the code that still reproduces the error but the problem shown up when it is called in a large, complex code.

[mmckerns]

Are you still using the same version of pathos? Here are the most recent relevant versions:

>>> import pathos
>>> pathos.__version__
'0.2.1.dev0'
>>> import dill
>>> dill.__version__
'0.2.6.dev0'
>>> import multiprocess
>>> multiprocess.__version__
'0.70.5.dev0'
>>> 

Your dill looks like it's 0.2.2, which is pretty old.

[minhhg]

I have fixed the error, thanks to your help "The typical reason for seeing the error you posted is that the function to be pickled has free variables that are referenced before assignment in enclosing scope". Actually some variables are not set on certain paths of execution. The code works fine when run in sequential mode but I guess when using pathos one needs to define all these variables even they are not needed for correct execution, is that right?

[mmckerns]

That's correct.