search

mrkn / pycall.rb

Calling Python functions from the Ruby language
MIT License
1.05k stars 72 forks source link

Using pycall on Dokku (Heroku compatible alternative) #146

Closed jankeesvw closed 3 years ago

jankeesvw commented 3 years ago

We are using pandas.rb in our development setup and we love it 🐼, I'm now trying to get this working on our production host. We use Dokku, which is a open source alternative to Heroku. It uses the same buildpack system, so I think pycall could just work.

When I run pycall locally I see this:

➜ rails c
Loading development environment (Rails 6.1.3.2)

irb(main):001:0> require 'pycall/import'
=> true
irb(main):002:0> include PyCall::Import
=> Object
irb(main):003:0> pyimport :math
=> :math
irb(main):004:0> math.sin(math.pi / 4) - Math.sin(Math::PI / 4)   # => 0.0
=> 0.0

Then I tried deploying this code, this is our requirements.txt file:

pandas
sqlalchemy

And this is the runtime.txt:

python-3.8.1

This is our .buildpacks file:

https://github.com/ReforgeHQ/heroku-buildpack-python
https://github.com/heroku/heroku-buildpack-ruby.git

When I do the same commands on our production environment I see this:

Loading staging environment (Rails 6.1.3.2)

irb(main):001:0> require 'pycall/import'
=> true
irb(main):002:0> include PyCall::Import
=> Object
irb(main):003:0> pyimport :math
Traceback (most recent call last):
        1: from (irb):3
PyCall::LibPythonFunctionNotFound (Unable to find the required symbol in libpython: _Py_NoneStruct)
irb(main):004:0> math.sin(math.pi / 4) - Math.sin(Math::PI / 4)   # => 0.0
irb(main):005:0>

Python is installed as you can see from this output:

Loading staging environment (Rails 6.1.3.2)
irb(main):001:0> `python3 --version`

=> "Python 3.8.1\n"
irb(main):002:0>

@mrkn you have any ideas on what this could be?

mrkn commented 3 years ago

I have no knowledge of Dokku. But, it would be helpful to guess the cause if you can give me the debug log of the libpython finder.

Could you try again with the instruction described here?

jankeesvw commented 3 years ago

Hi @mrkn, thanks for helping. When I try that I see this:

herokuishuser@ca6fcdc6ec7f:~$ ruby -rpycall -ePyCall.builtins
Traceback (most recent call last):
    5: from -e:1:in `<main>'
    4: from /app/vendor/bundle/ruby/2.7.0/gems/pycall-1.4.0/lib/pycall.rb:13:in `builtins'
    3: from /app/vendor/bundle/ruby/2.7.0/gems/pycall-1.4.0/lib/pycall/init.rb:16:in `const_missing'
    2: from /app/vendor/bundle/ruby/2.7.0/gems/pycall-1.4.0/lib/pycall/init.rb:39:in `init'
    1: from /app/vendor/ruby-2.7.3/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
/app/vendor/ruby-2.7.3/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': Unable to find the required symbol in libpython: _Py_NoneStruct (PyCall::LibPythonFunctionNotFound)

Does that make sense to you?

mrkn commented 3 years ago

I guess you didn't define PYCALL_DEBUG_FIND_LIBPYTHON environment variable likewise the instruction in README.

Could you please retry the investigation with the environment variable? I recommend you to copy-and-paste the command line in the instruction as is.

jankeesvw commented 3 years ago

That gives me this output:

herokuishuser@4fded4e73622:~$ ruby -rpycall -ePyCall.builtins
DEBUG(find_libpython) find_libpython(nil)
DEBUG(find_libpython) investigate_python_config("python3")
DEBUG(find_libpython) Candidate: /app/python3
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) libpaths: ["/app/.heroku/python/lib/python3.8/config-3.8-x86_64-linux-gnu", "/app/.heroku/python/lib", "/app/.heroku/python"]
DEBUG(find_libpython) candidate_names: ["libpython3.8.a", "libpython3.8.so", "libpython.so"]
DEBUG(find_libpython) Candidate: /app/.heroku/python/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a
DEBUG(find_libpython) Trying to dlopen: /app/.heroku/python/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a
DEBUG(find_libpython) dlopen("/app/.heroku/python/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a") => Fiddle::DLError: /app/.heroku/python/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a: invalid ELF header
DEBUG(find_libpython) Candidate: /app/.heroku/python/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: /app/.heroku/python/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython.so
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: /app/.heroku/python/lib/libpython3.8.a
DEBUG(find_libpython) Trying to dlopen: /app/.heroku/python/lib/libpython3.8.a
DEBUG(find_libpython) dlopen("/app/.heroku/python/lib/libpython3.8.a") => Fiddle::DLError: /app/.heroku/python/lib/libpython3.8.a: invalid ELF header
DEBUG(find_libpython) Candidate: /app/.heroku/python/lib/libpython3.8.so
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: /app/.heroku/python/lib/libpython.so
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: /app/.heroku/python/libpython3.8.a
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: /app/.heroku/python/libpython3.8.so
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: /app/.heroku/python/libpython.so
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: libpython3.8.a
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: libpython3.8.so
DEBUG(find_libpython) Not found.
DEBUG(find_libpython) Candidate: libpython.so
DEBUG(find_libpython) Not found.
Traceback (most recent call last):
    5: from -e:1:in `<main>'
    4: from /app/vendor/bundle/ruby/2.7.0/gems/pycall-1.4.0/lib/pycall.rb:13:in `builtins'
    3: from /app/vendor/bundle/ruby/2.7.0/gems/pycall-1.4.0/lib/pycall/init.rb:16:in `const_missing'
    2: from /app/vendor/bundle/ruby/2.7.0/gems/pycall-1.4.0/lib/pycall/init.rb:39:in `init'
    1: from /app/vendor/ruby-2.7.3/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
/app/vendor/ruby-2.7.3/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': Unable to find the required symbol in libpython: _Py_NoneStruct (PyCall::LibPythonFunctionNotFound)
mrkn commented 3 years ago

Could you check the output of the following commands?

which python3
ldd `which python3`
jankeesvw commented 3 years ago

This is the output: 

herokuishuser@4fded4e73622:~$ which python3
/app/.heroku/python/bin/python3
herokuishuser@4fded4e73622:~$ ldd `which python3`
    linux-vdso.so.1 (0x00007ffe0bde7000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa4d3128000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa4d2f24000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fa4d2d21000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa4d2983000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa4d2592000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa4d390a000)
mrkn commented 3 years ago

@jankeesvw I'm sorry for the response to be late.

Could you check the existence of libpython.so.3.8 in /app/.heroku/python and directories under that? I suspect that the python in /app/.heroku/python is not built with --enable-shared.

You need to specify where is python executable that is built with --enable-shared in the PYTHON environment variable, or add the path to such the python executable in the PATH environment variable.

jankeesvw commented 3 years ago

Thanks for the responses, we got it working by using these settings:

.buildpacks:

https://github.com/ReforgeHQ/heroku-buildpack-python.git
https://github.com/heroku/heroku-buildpack-ruby.git

runtime.txt

python-3.8.1
buncis commented 3 years ago

@jankeesvw sorry for asking in closed issue but I have same problem can I know heroku stack that you use?

I'm using heroku 20 with same config with yours but I still can't make it work

jankeesvw commented 3 years ago

@buncis I'm using Dokku, which is an alternative to Heroku (open source).

mkrumholz commented 3 years ago

@buncis did you get this figured out? I am having the same issue currently. I am not seeing how to configure python to enable shared libraries in my production environment (it runs perfectly on local)

buncis commented 3 years ago

yes I have solved this issue you could check my latest issue in this repo titled how to deploy to heroku

151 @mkrumholz

On Mon, Aug 2, 2021, 8:41 AM Molly Krumholz @.***> wrote:

@buncis https://github.com/buncis did you get this figured out? I am having the same issue currently. I am not seeing how to configure python to enable shared libraries in my production environment (it runs perfectly on local)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mrkn/pycall.rb/issues/146#issuecomment-890648785, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMZEWYHARMBLHTAJOR77XLT2XZU7ANCNFSM47GLGWGQ .

mkrumholz commented 3 years ago

@buncis THANK YOU SO MUCH! You fully just saved my project that I was despairing over. Completely made my day!! 🤩 🙏🏻 🎉