uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

cPickle.PicklingError python 2.7 #227

Closed gziras-in closed 2 years ago

gziras-in commented 2 years ago

Hello,

In a centos 7 server I try to run the following code.

from pathos.pools import ProcessPool as Pool
from time import sleep

def f(x):
    sleep(3)
    return x*x

if __name__ == '__main__':
    p = Pool(nodes=4)
    print(p.map(f, range(4)))

Trying to execute the above program I get: cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup builtin.function failed

Traceback (most recent call last): File "multi.py", line 14, in print(p.map(f, range(4))) File "lib/python/pathos-0.2.9.dev0-py2.7.egg/pathos/multiprocessing.py", line 139, in map return _pool.map(star(f), zip(*args)) # chunksize File "lib/python/multiprocess-0.70.12.2-py2.7.egg/multiprocess/pool.py", line 253, in map return self.map_async(func, iterable, chunksize).get() File "lib/python/multiprocess-0.70.12.2-py2.7.egg/multiprocess/pool.py", line 572, in get raise self._value cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup builtin.function failed

My stack:

**Python 2.7.5**
$ pip list | grep -e pathos -e ppft -e dill -e pox -e multiprocess -e six
dill (0.3.5.dev0)
multiprocess (0.70.12.2)
pathos (0.2.9.dev0)
pox (0.3.1.dev0)
ppft (1.6.6.5.dev0)
six (1.9.0)

By trying to install multiprocess I got also this warning.

WARNING: The C extensions could not be compiled
-----------------------------------------------------------------------

Maybe you do not have a C compiler installed on this system?
The reason was:
  File "setup.py", line 496, in <module>
    msg = BUILD_WARNING % '\n'.join(traceback.format_stack())

This is just a warning as most of the functionality will work even
without the updated C extension.  It will simply fallback to the
built-in _multiprocessing module.  Most notably you will not be able to use
FORCE_EXECV on POSIX systems.  If this is a problem for you then please
install a C compiler or fix the error(s) above.

But I have this: $ gcc --version gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)

Should I set an environment variable so that the compiler can be found?

Could you assist me with the above issue?

mmckerns commented 2 years ago

The C compiler needs be available in the path, and you'll need to have Python.h (and other relevant headers) installed. As you noted, this is likely a duplicate of #125.

gziras-in commented 2 years ago

yum install python-devel installed the necessary headers and then it worked.

Thank you!