compatibility with pypy

[imazor]

Hi,

I am trying to use pathos with pypy, however getting errors related to ctypes which is used by dill

Traceback (most recent call last): File "app_main.py", line 75, in run_toplevel File "adjust_logs_parser.py", line 17, in import pathos.multiprocessing as mp File "/usr/local/lib/python2.7/dist-packages/pathos-0.2a.dev_20130811-py2.7.egg/pathos/init.py", line 40, in import multiprocessing File "/usr/local/lib/python2.7/dist-packages/pathos-0.2a.dev_20130811-py2.7.egg/pathos/multiprocessing.py", line 69, in from pathos.helpers.mp_helper import starargs as star File "/usr/local/lib/python2.7/dist-packages/pathos-0.2a.dev_20130811-py2.7.egg/pathos/helpers/init.py", line 1, in import pp_helper File "/usr/local/lib/python2.7/dist-packages/pathos-0.2a.dev_20130811-py2.7.egg/pathos/helpers/pp_helper.py", line 12, in import dill as pickle File "/usr/local/lib/python2.7/dist-packages/dill/init.py", line 26, in from .dill import dump, dumps, load, loads, dump_session, load_session, \ File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 528, in ctypes.pythonapi.PyCell_New.restype = ctypes.py_object AttributeError: 'module' object has no attribute 'pythonapi'

is pathos not compatible with pypy ? is there workaround for the error ?

Thanks,

[mmckerns]

First thing I noticed it that you are using a slightly out of date version of pathos, and possibly a slightly incompatible version of dill. The best thing to do, for the moment (until I get the new release of pathos out), is to use the version of pathos from github. The version of pathos you are using is 1.5 yrs old. The version of dill may be old as well, as you should not be importing ctypes if it doesn't exist as a module. Now, I don't know if pypy has a ctypes module… if it does, then I'll need to add a patch for that case… and I'll need to see what methods it comes with.

Honestly, I don't know whether pathos is compatible with pypy… but it might be. Not sure about dill either. Can you post the version of dill you are using?

If pypy has ctypes, then I need to make a small patch to dill, where if pythonapi is not found, then it also skips the CellType and special cases for FileType. Then things should probably be fine. Actually, I should be complete… Let me know if any of these throw an error:

>>> import ctypes
>>> ctypes.pythonapi.PyCell_New
>>> ctypes.py_object
>>> ctypes.Structure
>>> ctypes.c_long 
>>> ctypes.c_voidp
>>> ctypes.c_int
>>> ctypes.POINTER
>>> ctypes.cast

dill uses these to support pickling file handles (in special cases) and also for pickling cell objects.

If the import of dill works after an appropriate ctypes patch, then pathos should also work.

[imazor]

Ok, so my version are the following (output from pip freeze)

pathos==0.2a1.dev - this one installed by build from source. I did git clone https://github.com/uqfoundation/pathos.git, following by python setup.py build and then python setup.py install. So have no idea how to get newer version. During the install I got some error messages, I think related to the pp:

Searching for pp>=1.6.4.4 Reading http://pypi.python.org/simple/pp/ Best match: pp 1.6.4.4 Downloading http://danse.cacr.caltech.edu/packages/dev_danse_us/pp-1.6.4.4.zip Processing pp-1.6.4.4.zip Writing /tmp/easy_install-iRTPsG/pp-1.6.4.4/setup.cfg Running pp-1.6.4.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-iRTPsG/pp-1.6.4.4/egg-dist-tmp-SORNVh Traceback (most recent call last): File "setup.py", line 346, in exec setup_code File "", line 249, in File "/usr/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/usr/local/lib/python2.7/dist-packages/setuptools/command/install.py", line 73, in run self.do_egg_install() File "/usr/local/lib/python2.7/dist-packages/setuptools/command/install.py", line 101, in do_egg_install cmd.run() File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 359, in run self.easy_install(spec, not self.no_deps) File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 583, in easy_install return self.install_item(None, spec, tmpdir, deps, True) File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 635, in install_item self.process_distribution(spec, dist, deps) File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 687, in process_distribution [requirement], self.local_index, self.easy_install File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 586, in resolve dist = best[req.key] = env.best_match(req, self, installer) File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 831, in best_match return self.obtain(req, installer) # try and download/install File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 843, in obtain return installer(requirement) File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 603, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 633, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 828, in install_eggs return self.build_and_install(setup_script, setup_base) File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1108, in build_and_install self.run_setup(setup_script, setup_base, args) File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1094, in run_setup run_setup(setup_script, args) File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 33, in run_setup lambda: execfile( File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 81, in run return func() File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 35, in {'file':setup_script, 'name':'main'} File "setup.py", line 23, in has_setuptools = False ImportError: cannot import name version

\ It would be really great if stable build via pip can be released (I think that it should install all the dependencies, such as dill, as well). I really think that issues with clear installation of the package can scarify a lot of potential users to explore your package. **

processing==0.52-pathos - this one installed via pip install pyre==0.8.2.0-pathos - this one was install from http://danse.cacr.caltech.edu/packages/dev_danse_us/ as pip install failed to find such package at all. dill==0.2.2 - this one installed via pip install, as far as I saw its the lat version in the repository as well. pox==0.2.1 - this one installed via pip install.

Regarding pypy: I have tried the different imports that you have mentioned, and the only one that fails is : ctypes.pythonapi.PyCell_New with the error message:

ctypes.pythonapi.PyCell_New Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'pythonapi'

Thanks,

On Tue, Jan 13, 2015 at 12:05 AM, Mike McKerns notifications@github.com wrote:

First thing I noticed it that you are using a slightly out of date version of pathos, and possibly a slightly incompatible version of dill. The best thing to do, for the moment (until I get the new release of pathos out), is to use the version of pathos from github. The version of pathos you are using is 1.5 yrs old. The version of dill may be old as well, as you should not be importing ctypes if it doesn't exist as a module. Now, I don't know if pypy has a ctypes module… if it does, then I'll need to add a patch for that case… and I'll need to see what methods it comes with.

Honestly, I don't know whether pathos is compatible with pypy… but it might be. Not sure about dill either. Can you post the version of dill you are using?

If pypy has ctypes, then I need to make a small patch to dill, where if pythonapi is not found, then it also skips the CellType and special cases for FileType. Then things should probably be fine. Actually, I should be complete… Let me know if any of these throw an error:

import ctypes ctypes.pythonapi.PyCell_New ctypes.py_object ctypes.Structure ctypes.c_long ctypes.c_voidp ctypes.c_int ctypes.POINTER ctypes.cast

dill uses these to support pickling file handles (in special cases) and also for pickling cell objects.

If the import of dill works after an appropriate ctypes patch, then pathos should also work.

— Reply to this email directly or view it on GitHub https://github.com/uqfoundation/pathos/issues/50#issuecomment-69665091.

--------------------------------------------------------------------------

Igor Mazor Senior Business Intelligence Manager

Rocket Internet AG | Johannisstraße 20 | 10117 Berlin | Deutschland skype: igor_rocket_internet | mail: igor.mazor http://goog_862328191 @rocket-internet.de www.rocket-internet.de

Geschäftsführer: Dr. Johannes Bruder, Arnt Jeschke, Alexander Kudlich Eingetragen beim Amtsgericht Berlin, HRB 109262 USt-ID DE256469659

[mmckerns]

Wow. I have no idea why the pp install fails. It should just be pulling __version__ out of ppcommon. You aren't using setuptools and also are installing pp through running setup on pathos… so that's a combination I didn't try. I'll see if I can track down the issue with that case.

I know that I tested the same install, but with setuptools, and it worked.

I know the installation is an issue… and one that needs work. The pathos package has a large dependency footprint, and most of the dependencies that have issues are forks of packages. I'll deal with all of that this month. I think I have a good clear plan that should resolve a bunch of these issues, but need the get the time to implement it.

Re the ctypes dependency: I'll make a new ticket for dill with this. I should be able to get the patch in for that immediately, and will probably do a patch release when pathos is released.

[mmckerns]

Since I don't test with pypy (I may do so in the near future, but I don't at the moment), could you let me know (on the dill issue ref above, or here for pathos issues) if there's further issues interacting with pypy? That is. once the install works.

[imazor]

sure, once you will release the patch for ctypes, I will be happy to test it again.

On Tue, Jan 13, 2015 at 2:43 PM, Mike McKerns notifications@github.com wrote:

Since I don't test with pypy (I may do so in the near future, but I don't at the moment), could you let me know (on the dill issue ref above, or here for pathos issues) if there's further issues interacting with pypy? That is. once the install works.

— Reply to this email directly or view it on GitHub https://github.com/uqfoundation/pathos/issues/50#issuecomment-69745311.

--------------------------------------------------------------------------

Igor Mazor Senior Business Intelligence Manager

Rocket Internet AG | Johannisstraße 20 | 10117 Berlin | Deutschland skype: igor_rocket_internet | mail: igor.mazor http://goog_862328191 @rocket-internet.de www.rocket-internet.de

Geschäftsführer: Dr. Johannes Bruder, Arnt Jeschke, Alexander Kudlich Eingetragen beim Amtsgericht Berlin, HRB 109262 USt-ID DE256469659

[mmckerns]

Ok, so I split off the issues, so the compatibility with ctypes is a dill issue (https://github.com/uqfoundation/dill/issues/73), and that the pp install failure is a new ticket #51. I'm going to leave this open, pending the dill fix. I don't foresee any other pathos compatibility issues with pypy, but I also don't know if I'm forgetting some cases in some of the other dependencies.

[mmckerns]

It seems as if dill is "fixed" (at least zeroth-order) for pypy compatibility, however there now appears to be pathos compatibility issues (as noted in the issue ref'd above). Will continue the issue here, as there appear to be direct pathos issues.

The meat of the referenced traceback is:

"/usr/local/lib/python2.7/dist-packages/pathos-0.2a1.dev-py2.7.egg/pathos/multiprocessing.py", line 123, in map
    return _pool.map(star(f), zip(*args)) # chunksize
File "/var/opt/pypy/lib-python/2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
File "/var/opt/pypy/lib-python/2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
AttributeError: 'function' object has no attribute '__objclass__'

[mmckerns]

should be some progress on this, due to #53, but still have not taken care of #54.

[mmckerns]

Both #53 and #54 were fixed a while back, and now that https://github.com/uqfoundation/dill/issues/73 is also fixed, it seems this should also be closed. All pathos tests pass with pypy.