search

mrkn / pycall.rb

Calling Python functions from the Ruby language
MIT License
1.06k stars 75 forks source link

(on 1.3.0): Fatal Python error: Py_Initialize: Unable to get the locale encoding #121

Open IwanVosloo opened 4 years ago

IwanVosloo commented 4 years ago

Hi, we're using pycall 1.3.0, with python 3.5.2 from a venv and ruby 2.6.3p62.

We're getting this error when running PYCALL_DEBUG_FIND_LIBPYTHON=1 ruby -rpycall -ePyCall.builtins

DEBUG(find_libpython) find_libpython(nil)
DEBUG(find_libpython) investigate_python_config("python3")
DEBUG(find_libpython) libs: ["libpython3.5m.so.1.0", "libpython3.5m.so", "libpython3.5m", "libpython3.5", "libpython"]
DEBUG(find_libpython) libpaths: ["/usr/lib", "/var/local/venv/development/lib", "", "/var/local/venv/development", "/var/local/venv/development/lib"]
DEBUG(find_libpython) Unable to find /usr/lib/libpython3.5m.so.1.0
DEBUG(find_libpython) Unable to find /usr/lib/libpython3.5m.so.1.0.so
DEBUG(find_libpython) dlopen("/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0") = #<Fiddle::Handle:0x000055e513240bf0>
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Current thread 0x00007f71fb68d700 (most recent call first):
Aborted (core dumped)

With pycall 1.2.1 in the same setup we don't have the problem.

mrkn commented 4 years ago

That error has occurred in Python-side, so it seems not pycall.rb's issue. Please investigate your Python's settings such as environment variables.

IwanVosloo commented 4 years ago

Hi!

On 2020/05/25 04:41, Kenta Murata wrote:

That error has occurred in Python-side, so it seems not pycall.rb's issue. Please investigate your Python's settings such as environment variables.

Can you please explain your reasoning? If everything works fine with pycall 1.2.1, and just upgrading to pycall 1.3.0 causes the problem its hard for me to understand that the error is outside of what pycall does.

Sure, it might have to do with PYTHONPATH or other variables controlling python, but I was under the impression its pycall's taks to set these up and invoke python correctly?

Is there something I can check and post here that can throw more light on the issue?

Regards Iwan

--

mrkn commented 4 years ago

I missed looking at the last line of the description. I'm sorry.

I've tried to reproduce this with Python 3.5.9, but I couldn't install Python 3.5.9 in my macOS due to the build error. Please wait some days for constructing the testing environment.

By the way, how do you construct your environment? I noticed your libpaths contains the string venv:

DEBUG(find_libpython) libpaths: ["/usr/lib", "/var/local/venv/development/lib", "", "/var/local/venv/development", "/var/local/venv/development/lib"]

If you didn't use the vanilla environment of Python, please tell me how to prepare the environment that is the same as yours.

IwanVosloo commented 4 years ago

Ah, thanks for looking into it - I'd have no idea where to look!

This is on stock ubuntu 16.04, with its stock python 3.5.2 installed (called python3 on ubuntu 16.04). We also install stock python3-pip and python3-venv.

The path /var/local/venv/development you see is the path to a venv, created (assuming /var/local/venv exists) by running:

pyvenv /var/local/venv/development

We also install inside the venv: pip==20.1.1 setuptools==46.4.0

I have also now installed it on Ubuntu 18.04 with the same results, here I used: python3 -m venv /var/local/venv/development to create the venv

And I used the stock python3, python2-pip, python3-venv of ubuntu 18.04 with the same pip and setuptools as above.

Thanks Iwan

On 2020/05/27 00:45, Kenta Murata wrote:

I'm sorry, I missed the last line of the description.

I've tried to reproduce this with Python 3.5.9, but I couldn't install Python 3.5.9 in my macOS due to the build error. Please wait some days for constructing the testing environment.

By the way, how do you construct your environment? I noticed your libpaths contains the string |venv|:

|DEBUG(find_libpython) libpaths: ["/usr/lib",
"/var/local/venv/development/lib", "",
"/var/local/venv/development", "/var/local/venv/development/lib"] |

If you didn't use the vanilla environment of Python, please tell me how to prepare the environment that is the same as yours.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mrkn/pycall.rb/issues/121#issuecomment-634319900, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARGTN247HKQMROO5SDORNDRTRBCHANCNFSM4NDACSLA.

--

Reahl, the Python only web framework / www.reahl.org

holyketzer commented 4 years ago

Have same error on Mac OS through venv. Not elegant but using system Python solves this

jmarrec commented 3 years ago

I'm having a similar issue on ubuntu 18.04, on a virtualenv, with 1.3.0 and 1.3.1. 1.2.1 works (but complains about undefined methodwithout_gvl' for PyCall:Module`)

DEBUG(find_libpython) find_libpython(nil)
DEBUG(find_libpython) investigate_python_config("python3")
DEBUG(find_libpython) libs: ["libpython3.8.so.1.0", "libpython3.8.so", "libpython3.8", "libpython"]
DEBUG(find_libpython) libpaths: ["/home/julien/.pyenv/versions/3.8.3/lib", "/home/julien/Virtualenvs/py38/lib", "", "/home/julien/Virtualenvs/py38", "/home/julien/Virtualenvs/py38/lib"]
DEBUG(find_libpython) dlopen("/home/julien/.pyenv/versions/3.8.3/lib/libpython3.8.so.1.0") = #<Fiddle::Handle:0x000055d6c0a02ac0>
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 769, in <module>
    main()
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 746, in main
    paths_in_sys = addsitepackages(paths_in_sys)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 279, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 202, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 170, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/_virtualenv.py", line 41, in <module>
    from importlib.util import find_spec
  File "/home/julien/Virtualenvs/py38/lib/python3.8/importlib/util.py", line 14, in <module>
    from contextlib import contextmanager
ModuleNotFoundError: No module named 'contextlib'
mrkn commented 3 years ago

@jmarrec Could you please explain the steps for reproducing the same problem you encountered?

jmarrec commented 3 years ago

@mrkn Yes sure, thanks for your prompt reply.

Install pycall 1.2.1

(py38)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ gem install pycall -v 1.2.1
Fetching pycall-1.2.1.gem
iBuilding native extensions. This could take a while...
rbSuccessfully installed pycall-1.2.1
1 gem installed

Require 'sys', it works:

(py38)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ irb
2.5.5 :001 > require 'pycall/import'
 => true 
2.5.5 :002 > include PyCall::Import
 => Object 
2.5.5 :003 > pyimport 'sys'
 => :sys

Now uninstall pycall 1.2.1, and install 1.3.1 instead:

(py38)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ gem uninstall pycall
Successfully uninstalled pycall-1.2.1
(py38)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ gem install pycall
Fetching pycall-1.3.1.gem
Building native extensions. This could take a while...
Successfully installed pycall-1.3.1
1 gem installed

And try again:

(py38)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ irb
2.5.5 :001 > require 'pycall/import'
 => true 
2.5.5 :002 > include PyCall::Import
 => Object 
2.5.5 :003 > pyimport 'sys'
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 769, in <module>
    main()
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 746, in main
    paths_in_sys = addsitepackages(paths_in_sys)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 279, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 202, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 170, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/_virtualenv.py", line 41, in <module>
    from importlib.util import find_spec
  File "/home/julien/Virtualenvs/py38/lib/python3.8/importlib/util.py", line 14, in <module>
    from contextlib import contextmanager
ModuleNotFoundError: No module named 'contextlib'

To install my python environment, I used pyenv with the following options (I need shared libs for some embedding)

PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.8.3 (at the time it was the highest available)

I then create a virtualenv env using this pyenv python via mkvirtualenv py38 --system-site-packages

(I also have the same exact "problem" with 3.9.1)

I actually have my own markdown file for setting it all up, I'm sharing it here in case it's useful

How to install pyenv and virtualenv ## Install build requirements See pyenv's wiki [here](https://github.com/pyenv/pyenv/wiki/common-build-problems#requirements) for an up to date list: sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ xz-utils tk-dev libffi-dev liblzma-dev python-openssl ## Install pyenv https://github.com/pyenv/pyenv-installer: **Note**: uses pyenv-virtualenv and not pyenv-virtualenvwrapper $ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash or, directly from https://github.com/pyenv/pyenv#installation, Basically step1 only: $ git clone git@github.com:pyenv/pyenv.git ~/.pyenv ## Install a python version via pyenv To avoid getting errors like this when linking E+: ``` relocation R_X86_64_PC32 against symbol `_PyRuntime' can not be used when making a shared object; recompile with -fPIC ``` use `PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install X.Y.Z` # List all available versions $ pyenv install --list # Install one $ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.1 # Use that $ pyenv local 3.9.1 $ pyenv global 3.9.1 **Do verify that `python --version` gives out 3.9.1 before proceeding** ## Pyenv-virtualenvwrapper as a pyenv plugin https://github.com/pyenv/pyenv-virtualenvwrapper $ git clone git@github.com:pyenv/pyenv-virtualenvwrapper.git $(pyenv root)/plugins/pyenv-virtualenvwrapper ## Create a python venv mkvirtualenv py39 --system-site-packages
jmarrec commented 3 years ago

some environment variables that may be of interest:

(py38)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ echo $PYTHONHOME

(py38)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ echo $PYTHONPATH

(py38)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ echo $PATH
/home/julien/Virtualenvs/py38/bin:/home/julien/.pyenv/plugins/pyenv-virtualenvwrapper/shims:/home/julien/.pyenv/libexec:/home/julien/.pyenv/plugins/python-build/bin:/home/julien/.pyenv/plugins/pyenv-virtualenvwrapper/bin:/home/julien/.pyenv/plugins/python-build/bin:/home/julien/.pyenv/plugins/pyenv-virtualenvwrapper/bin:/home/julien/.pyenv/shims:~/.pyenv/bin:/home/julien/Qt/QtIFW-4.0.0/bin/:~/.config/composer/vendor/bin:/home/julien/.local/bin:/home/julien/.rvm/gems/ruby-2.5.5/bin:/home/julien/.rvm/gems/ruby-2.5.5@global/bin:/home/julien/.rvm/rubies/ruby-2.5.5/bin:/home/julien/.rvm/gems/ruby-2.5.5/bin:/home/julien/.rvm/gems/ruby-2.5.5@global/bin:/home/julien/.rvm/rubies/ruby-2.5.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/julien/.dotnet/tools:/home/julien/.rvm/bin:/home/julien/.dotnet/tools
jmarrec commented 3 years ago

I can also reproduce with the same command as the OP:

$ PYCALL_DEBUG_FIND_LIBPYTHON=1 ruby -rpycall -ePyCall.builtins

DEBUG(find_libpython) find_libpython(nil)
DEBUG(find_libpython) investigate_python_config("python3")
DEBUG(find_libpython) libs: ["libpython3.8.so.1.0", "libpython3.8.so", "libpython3.8", "libpython"]
DEBUG(find_libpython) libpaths: ["/home/julien/.pyenv/versions/3.8.3/lib", "/home/julien/Virtualenvs/py38/lib", "", "/home/julien/Virtualenvs/py38", "/home/julien/Virtualenvs/py38/lib"]
DEBUG(find_libpython) dlopen("/home/julien/.pyenv/versions/3.8.3/lib/libpython3.8.so.1.0") = #<Fiddle::Handle:0x00005577803400d8>
paths_in_sys={'/home/julien/Virtualenvs/py38/lib/python3.8', '/home/julien/Virtualenvs/py38/lib/python38.zip', '/home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall/python', '/home/julien/.pyenv/versions/3.8.0/lib/python3.8', '/home/julien/Virtualenvs/py38/lib/python3.8/lib-dynload'}
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 770, in <module>
    main()
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 747, in main
    paths_in_sys = addsitepackages(paths_in_sys)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 279, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 202, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site.py", line 170, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/_virtualenv.py", line 41, in <module>
    from importlib.util import find_spec
  File "/home/julien/Virtualenvs/py38/lib/python3.8/importlib/util.py", line 14, in <module>
    from contextlib import contextmanager
ModuleNotFoundError: No module named 'contextlib'
jmarrec commented 3 years ago

Ok, if I do that with python 3.9, which wraps stuff in Lib/site.py addpackage method in a try except block, I get the same fs_encoding error as the OP:

(py39)julien@OpenStudio-workflow-gem (python_measure_mod *>)$ PYCALL_DEBUG_FIND_LIBPYTHON=1 ruby -rpycall -ePyCall.builtins
DEBUG(find_libpython) find_libpython(nil)
DEBUG(find_libpython) investigate_python_config("python3")
DEBUG(find_libpython) libs: ["libpython3.9.so.1.0", "libpython3.9.so", "libpython3.9", "libpython"]
DEBUG(find_libpython) libpaths: ["/home/julien/.pyenv/versions/3.9.1/lib", "/home/julien/Virtualenvs/py39/lib", "", "/home/julien/Virtualenvs/py39", "/home/julien/Virtualenvs/py39/lib"]
DEBUG(find_libpython) dlopen("/home/julien/.pyenv/versions/3.9.1/lib/libpython3.9.so.1.0") = #<Fiddle::Handle:0x000055d0670afd10>
Python path configuration:
  PYTHONHOME = '/home/julien/Virtualenvs/py39:/home/julien/Virtualenvs/py39'
  PYTHONPATH = '/home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall/python'
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/home/julien/Virtualenvs/py39/bin/python3'
  sys.base_prefix = '/home/julien/Virtualenvs/py39'
  sys.base_exec_prefix = '/home/julien/Virtualenvs/py39'
  sys.platlibdir = 'lib'
  sys.executable = '/home/julien/Virtualenvs/py39/bin/python3'
  sys.prefix = '/home/julien/Virtualenvs/py39'
  sys.exec_prefix = '/home/julien/Virtualenvs/py39'
  sys.path = [
    '/home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall/python',
    '/home/julien/Virtualenvs/py39/lib/python39.zip',
    '/home/julien/Virtualenvs/py39/lib/python3.9',
    '/home/julien/Virtualenvs/py39/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fcfd6013740 (most recent call first):
<no Python frame>
jmarrec commented 3 years ago

In my 3.8.3 version if I modify this: https://github.com/python/cpython/blob/6f8c8320e9eac9bc7a7f653b43506e75916ce8e8/Lib/site.py#L169-L170

by 

                try:
                    exec(line)
                    continue
                except:
                    print(f"addpackage, failed for {fullname=}")
                    print(f"exec(line): {line=}")
                    print("\n")
                    continue

Then it loads the sys correctly.

Here is the output:

$ PYCALL_DEBUG_FIND_LIBPYTHON=1 ruby -rpycall -ePyCall.builtins
DEBUG(find_libpython) find_libpython(nil)
DEBUG(find_libpython) investigate_python_config("python3")
DEBUG(find_libpython) libs: ["libpython3.8.so.1.0", "libpython3.8.so", "libpython3.8", "libpython"]
DEBUG(find_libpython) libpaths: ["/home/julien/.pyenv/versions/3.8.3/lib", "/home/julien/Virtualenvs/py38/lib", "", "/home/julien/Virtualenvs/py38", "/home/julien/Virtualenvs/py38/lib"]
DEBUG(find_libpython) dlopen("/home/julien/.pyenv/versions/3.8.3/lib/libpython3.8.so.1.0") = #<Fiddle::Handle:0x00005626ae2a61d0>
addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/_virtualenv.pth'
exec(line): line='import _virtualenv'

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/google_api_core-1.23.0-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])));m = m or sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/google_auth-1.23.0-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])));m = m or sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/googleapis_common_protos-1.52.0-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])));m = m or sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/googleapis_common_protos-1.52.0-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])));m = m or sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/googleapis_common_protos-1.52.0-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google', 'logging'));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google.logging', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google.logging', [os.path.dirname(p)])));m = m or sys.modules.setdefault('google.logging', types.ModuleType('google.logging'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p);m and setattr(sys.modules['google'], 'logging', m)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/matplotlib-3.3.3-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('mpl_toolkits',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('mpl_toolkits', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('mpl_toolkits', [os.path.dirname(p)])));m = m or sys.modules.setdefault('mpl_toolkits', types.ModuleType('mpl_toolkits'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/protobuf-3.13.0-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])));m = m or sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/sphinxcontrib_applehelp-1.0.2-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('sphinxcontrib',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('sphinxcontrib', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('sphinxcontrib', [os.path.dirname(p)])));m = m or sys.modules.setdefault('sphinxcontrib', types.ModuleType('sphinxcontrib'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/sphinxcontrib_devhelp-1.0.2-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('sphinxcontrib',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('sphinxcontrib', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('sphinxcontrib', [os.path.dirname(p)])));m = m or sys.modules.setdefault('sphinxcontrib', types.ModuleType('sphinxcontrib'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/sphinxcontrib_htmlhelp-1.0.3-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('sphinxcontrib',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('sphinxcontrib', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('sphinxcontrib', [os.path.dirname(p)])));m = m or sys.modules.setdefault('sphinxcontrib', types.ModuleType('sphinxcontrib'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/sphinxcontrib_jsmath-1.0.1-py3.7-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('sphinxcontrib',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('sphinxcontrib', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('sphinxcontrib', [os.path.dirname(p)])));m = m or sys.modules.setdefault('sphinxcontrib', types.ModuleType('sphinxcontrib'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/sphinxcontrib_qthelp-1.0.3-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('sphinxcontrib',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('sphinxcontrib', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('sphinxcontrib', [os.path.dirname(p)])));m = m or sys.modules.setdefault('sphinxcontrib', types.ModuleType('sphinxcontrib'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/sphinxcontrib_serializinghtml-1.1.4-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('sphinxcontrib',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('sphinxcontrib', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('sphinxcontrib', [os.path.dirname(p)])));m = m or sys.modules.setdefault('sphinxcontrib', types.ModuleType('sphinxcontrib'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

addpackage, failed for fullname='/home/julien/Virtualenvs/py38/lib/python3.8/site-packages/sphinxcontrib_websupport-1.2.4-py3.8-nspkg.pth'
exec(line): line="import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('sphinxcontrib',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('sphinxcontrib', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('sphinxcontrib', [os.path.dirname(p)])));m = m or sys.modules.setdefault('sphinxcontrib', types.ModuleType('sphinxcontrib'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)\n"

but it seems that it doesn't really work in real applications. One of my pycall call's logs this

[11:59:29.449264 ERROR] /home/julien/Software/Others/openstudio_gems/OpenStudio-workflow-gem/lib/openstudio/workflow/util/measure.rb failed with message <class 'ModuleNotFoundError'>: No module named 'traceback' in /home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall.rb:62:in `import_module'
/home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall.rb:62:in `import_module'
/home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall/pyerror.rb:27:in `format_traceback'
/home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall/pyerror.rb:16:in `block in to_s'
/home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall/pyerror.rb:15:in `tap'
/home/julien/.rvm/gems/ruby-2.5.5/gems/pycall-1.3.1/lib/pycall/pyerror.rb:15:in `to_
jmarrec commented 3 years ago

@mrkn I did a git bisect, each time doing gem uninstall pycall && bundle exec rake install && PYCALL_DEBUG_FIND_LIBPYTHON=1 ruby -rpycall -ePyCall.builtins and indicating I would see the failed imports messages or not.

2752e2e85a328bdbf251e859b902deaac4b2c182 is the first bad commit

jmarrec commented 3 years ago

I opened #135 as a result.

monsha commented 3 years ago

Having similar issue on a virtualenv, using pandas directly in virtualenv python works, but when I try to import pandas in ruby I get different errors.

Python 3.9.1 pandas-1.2.2 Ruby 2.5.3

With pycall 1.3.1 and 1.3.0:

2.5.3 :001 > require "pycall/import"
 => true 
2.5.3 :003 > include PyCall::Import
 => Object 
2.5.3 :005 > pyimport "pandas"
 Python path configuration:
  PYTHONHOME = '/Users/natasha/Projects/cm/env:/Users/natasha/Projects/cm/env'
  PYTHONPATH = '/Users/natasha/.rvm/gems/ruby-2.5.3/gems/pycall-1.3.0/lib/pycall/python'
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/Users/natasha/.rvm/rubies/ruby-2.5.3/bin/ruby'
  sys.base_prefix = '/Users/natasha/Projects/cm/env'
  sys.base_exec_prefix = '/Users/natasha/Projects/cm/env'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/natasha/.rvm/rubies/ruby-2.5.3/bin/ruby'
  sys.prefix = '/Users/natasha/Projects/cm/env'
  sys.exec_prefix = '/Users/natasha/Projects/cm/env'
  sys.path = [
    '/Users/natasha/.rvm/gems/ruby-2.5.3/gems/pycall-1.3.0/lib/pycall/python',
    '/Users/natasha/Projects/cm/env/lib/python39.zip',
    '/Users/natasha/Projects/cm/env/lib/python3.9',
    '/Users/natasha/Projects/cm/env/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

With 1.2.1

2.5.3 :001 > require "pycall/import"
 => true 
2.5.3 :003 > include PyCall::Import
 => Object 
2.5.3 :005 > pyimport :pandas
Traceback (most recent call last):
        5: from /Users/natasha/.rvm/rubies/ruby-2.5.3/bin/irb:11:in `<main>'
        4: from (irb):3
        3: from /Users/natasha/.rvm/gems/ruby-2.5.3@cm/gems/pycall-1.2.1/lib/pycall/import.rb:18:in `pyimport'
        2: from /Users/natasha/.rvm/gems/ruby-2.5.3@cm/gems/pycall-1.2.1/lib/pycall.rb:62:in `import_module'
        1: from /Users/natasha/.rvm/gems/ruby-2.5.3@cm/gems/pycall-1.2.1/lib/pycall.rb:62:in `import_module'
PyCall::PyError (<class 'ModuleNotFoundError'>: No module named 'pandas')