search

mrkn / pycall.rb

Calling Python functions from the Ruby language
MIT License
1.06k stars 75 forks source link

Weird path on anaconda3-2019.07 installed with pyenv #99

Closed rmatsumiya closed 5 years ago

rmatsumiya commented 5 years ago

I'd like to use PyCall with anaconda installed with pyenv. However, I caught an error whenever importing a module with pyimport.

>> require 'pycall'
=> true
>> require 'pycall/import'
=> true
>> include PyCall::Import
=> Object
>> pyimport 'sys'
/home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/libpython/finder.rb:102: warning: Insecure world writable dir /home/XXXXX/.gem/ruby/2.6.3/bin in PATH, mode 040777
Traceback (most recent call last):
  File "/home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/python/investigator.py", line 1, in <module>
    from distutils.sysconfig import get_config_var
ModuleNotFoundError: No module named 'distutils.sysconfig'
PyCall::PythonNotFound (PyCall::PythonNotFound)
    from /home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/libpython/finder.rb:96:in `find_libpython'
    from /home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/init.rb:35:in `init'
    from /home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/init.rb:16:in `const_missing'
    from /home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall.rb:62:in `import_module'
    from /home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/import.rb:18:in `pyimport'
    from (irb):4

When I manually initialize PyCall with PyCall.init, some modules like sys can be imported. However, some modules like scipy and numpy cannot be imported yet.

>> require 'pycall'
=> true
>> require 'pycall/import'
=> true
>> PyCall.init '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/bin/python'
/home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/libpython/finder.rb:102: warning: Insecure world writable dir /home/XXXXX/.gem/ruby/2.6.3/bin in PATH, mode 040777
=> true
>> include PyCall::Import
=> Object
>> pyimport 'sys'
=> :sys
>> sys.path
=> ['/home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/python', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python37.zip', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages']
>> pyimport 'scipy'
PyCall::PyError (<class 'ModuleNotFoundError'>: No module named '_ctypes')
  File "/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages/scipy/__init__.py", line 62, in <module>
    from numpy import show_config as show_numpy_config
  File "/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages/numpy/__init__.py", line 140, in <module>
    from . import _distributor_init
  File "/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages/numpy/_distributor_init.py", line 33, in <module>
    with RTLD_for_MKL():
  File "/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages/numpy/_distributor_init.py", line 18, in __enter__
    import ctypes
  File "/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
    from /home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall.rb:62:in `import_module'
    from /home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall.rb:62:in `import_module'
    from /home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.2.1/lib/pycall/import.rb:18:in `pyimport'
    from (irb):7

Of course, the native python can load such modules normally.

$  /home/XXXXX/.pyenv/versions/anaconda3-2019.07/bin/python
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python37.zip', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/lib-dynload', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages']
>>> import scipy
>>>

As you can see, the path for lib-dynload directory of native python and that of PyCall are different although other paths are identical. I suspect it is related to this problem.

mrkn commented 5 years ago

@rmatsumiya I couldn't reproduce the issue on my MacBook Pro.

$ which python
/opt/brew/bin/python
$ pyenv
-bash: pyenv: command not found
$ # The above two commands show pyenv wasn't configured.
$ irb
irb(main):001:0> require "pycall"
=> true
irb(main):002:0> PyCall.init("/Users/mrkn/.pyenv/versions/anaconda3-2019.07/bin/python")
=> true
irb(main):003:0> require "pycall/import"
=> true
irb(main):004:0> include PyCall::Import
=> Object
irb(main):005:0> pyimport "sys"
=> :sys
irb(main):006:0> sys.path
=> ['/Users/mrkn/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/pycall-1.2.1/lib/pycall/python', '/Users/mrkn/.pyenv/versions/a
naconda3-2019.07/lib/python37.zip', '/Users/mrkn/.pyenv/versions/anaconda3-2019.07/lib/python3.7', '/Users/mrkn/.pyenv/versions
/anaconda3-2019.07/lib/python3.7/lib-dynload', '/Users/mrkn/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages', '/U
sers/mrkn/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages/aeosa']
irb(main):007:0> pyimport "scipy"
=> :scipy
irb(main):008:0> pyimport "numpy"
=> :numpy
irb(main):009:0> numpy.linspace(0, 10, 50)
=> array([ 0.        ,  0.20408163,  0.40816327,  0.6122449 ,  0.81632653,
        1.02040816,  1.2244898 ,  1.42857143,  1.63265306,  1.83673469,
        2.04081633,  2.24489796,  2.44897959,  2.65306122,  2.85714286,
        3.06122449,  3.26530612,  3.46938776,  3.67346939,  3.87755102,
        4.08163265,  4.28571429,  4.48979592,  4.69387755,  4.89795918,
        5.10204082,  5.30612245,  5.51020408,  5.71428571,  5.91836735,
        6.12244898,  6.32653061,  6.53061224,  6.73469388,  6.93877551,
        7.14285714,  7.34693878,  7.55102041,  7.75510204,  7.95918367,
        8.16326531,  8.36734694,  8.57142857,  8.7755102 ,  8.97959184,
        9.18367347,  9.3877551 ,  9.59183673,  9.79591837, 10.        ])
irb(main):010:0>

Note that the anaconda environment used in the above experiment was installed by pycall install anaconda-3-2019.07.

mrkn commented 5 years ago

@rmatsumiya I can try again this on my Linux environment in my office next Tuesday. If you can know additional conditions that are necessary to reproduce this, please tell me.

rmatsumiya commented 5 years ago

@mrkn I have found a work-around for this issue. The /usr/lib/python3.7/lib-dynload is created by Ubuntu default installation for python3-gdbm. https://packages.ubuntu.com/bionic-updates/python3-gdbm This packages is depended by command-not-found (via python3-commandnotfound) https://packages.ubuntu.com/bionic/command-not-found

I uninstalled command-not-found and python3-gdbm with apt, then the lib-dynload directory is removed automatically. Finally, the sys.path seems to be fixed, and scipy and numpy have been correctly imported with PyCall.

We can conclude traversing python library directories might point unexpected directories and this leads to fail to import some packages. I guess we need to know where the root cause is; in the Python runtime, in PyCall, in pyenv, etc.

rmatsumiya commented 5 years ago

@mrkn I found this anaconda-detection no longer works. https://github.com/mrkn/pycall.rb/blob/f7ec9f8e788178030531a5f23bc37c0dd36f8b4c/lib/pycall/python/investigator.py#L5

In my investigation, sys.version does not include Continuum if using this version of anaconda.

$ python
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.7.3 (default, Mar 27 2019, 22:11:17) \n[GCC 7.3.0]'
rmatsumiya commented 5 years ago

@mrkn I posted a PR to close our problem (#100).

The root cause is the anaconda detection logic as above. As you may know, Continuum Analytics, Inc. was renamed Anaconda, Inc. So, the current implementation of conda checking depends on the anaconda version.

Fortunately, I found the logic is not really necessary by replacing path configuration after LIBPYTHON configuration.

In my environment, my branch fixes the problem completely without uninstalling python3-gdbm.

$ ls /usr/lib/python3.7/lib-dynload/
_gdbm.cpython-37m-x86_64-linux-gnu.so
$
>> require 'pycall'
=> true
>> PyCall.init '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/bin/python'
/home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.3.0.dev/lib/pycall/libpython/finder.rb:101: warning: Insecure world writable dir /home/XXXXX/.gem/ruby/2.6.3/bin in PATH, mode 040777
=> true
>> require 'pycall/import'
=> true
>> include PyCall::Import
=> Object
>> pyimport 'sys'
=> :sys
>> sys.path
=> ['/home/XXXXX/.gem/ruby/2.6.3/gems/pycall-1.3.0.dev/lib/pycall/python', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python37.zip', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/lib-dynload', '/home/XXXXX/.pyenv/versions/anaconda3-2019.07/lib/python3.7/site-packages']
>> pyimport 'scipy'
=> :scipy
>> pyimport 'numpy'
=> :numpy
>> numpy.linspace(0, 10, 50)
=> array([ 0.        ,  0.20408163,  0.40816327,  0.6122449 ,  0.81632653,
        1.02040816,  1.2244898 ,  1.42857143,  1.63265306,  1.83673469,
        2.04081633,  2.24489796,  2.44897959,  2.65306122,  2.85714286,
        3.06122449,  3.26530612,  3.46938776,  3.67346939,  3.87755102,
        4.08163265,  4.28571429,  4.48979592,  4.69387755,  4.89795918,
        5.10204082,  5.30612245,  5.51020408,  5.71428571,  5.91836735,
        6.12244898,  6.32653061,  6.53061224,  6.73469388,  6.93877551,
        7.14285714,  7.34693878,  7.55102041,  7.75510204,  7.95918367,
        8.16326531,  8.36734694,  8.57142857,  8.7755102 ,  8.97959184,
        9.18367347,  9.3877551 ,  9.59183673,  9.79591837, 10.        ])
mrkn commented 5 years ago

@rmatsumiya Could you please examine the the latest master branch?

rmatsumiya commented 5 years ago

I'm sorry for late response.

This problem seems to be fixed with the latest master branch, thank you for your cooperation.