search

emk / heroku-buildpack-rust

A buildpack for Rust applications on Heroku, with full support for Rustup, cargo and build caching.
522 stars 186 forks source link

Ruby/Rust ld cannot find lruby #15

Closed elifoster closed 4 years ago

elifoster commented 8 years ago

I've built a Ruby/Rust hello application, but it does not want to deploy to Heroku. @schneems 

remote:        Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
remote:        
remote:        current directory: /tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ex
remote:        /tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/ruby-2.3.0/bin/ruby -r ./siteconf20160926-228-cc92i.rb extconf.rb
remote:        cargo 0.12.0-nightly (6b98d1f 2016-07-04)
remote:        rustc 1.11.0 (9b21dcd6a 2016-08-15)
remote:        
remote:        current directory: /tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ex
remote:        make "DESTDIR=" clean
remote:        rm -rf target
remote:        
remote:        current directory: /tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ex
remote:        make "DESTDIR="
remote:        cargo build --release
remote:        Updating registry `https://github.com/rust-lang/crates.io-index`
remote:        Downloading ruru v0.7.8
remote:        Downloading ruby-sys v0.2.13
remote:        Downloading libc v0.2.16
remote:        Compiling ruby-sys v0.2.13
remote:        Compiling libc v0.2.16
remote:        Compiling ruru v0.7.8
remote:        Compiling string-utility-rb-native v1.0.0 (file:///tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ext)
remote:        src/lib.rs:4:12: 4:21 warning: unused import, #[warn(unused_imports)] on by default
remote:        src/lib.rs:4 use ruru::{AnyObject, Class, RString, VM};
remote:        ^~~~~~~~~
remote:        src/lib.rs:4:39: 4:41 warning: unused import, #[warn(unused_imports)] on by default
remote:        src/lib.rs:4 use ruru::{AnyObject, Class, RString, VM};
remote:        ^~
remote:        src/lib.rs:5:19: 5:23 warning: unused import, #[warn(unused_imports)] on by default
remote:        src/lib.rs:5 use ruru::types::{Argc, Value};
remote:        ^~~~
remote:        src/lib.rs:5:25: 5:30 warning: unused import, #[warn(unused_imports)] on by default
remote:        src/lib.rs:5 use ruru::types::{Argc, Value};
remote:        ^~~~~
remote:        src/lib.rs:6:5: 6:25 warning: unused import, #[warn(unused_imports)] on by default
remote:        src/lib.rs:6 use ruru::traits::Object;
remote:        ^~~~~~~~~~~~~~~~~~~~
remote:        error: linking with `cc` failed: exit code: 1
remote:        note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/app/tmp/cache/multirust/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ext/target/release/stringutility.0.o" "-o" "/tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ext/target/release/libstringutility.so" "/tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ext/target/release/stringutility.metadata.o" "-Wl,-O1" "-nodefaultlibs" "-L" "/tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ext/target/release" "-L" "/tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3/ext/target/release/deps" "-L" "/app/vendor/ruby-2.3.1/lib" "-L" "/app/tmp/cache/multirust/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/libruru-35c12ee1cec9de02.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/libruby_sys-a3c0096998cfe6fb.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/liblibc-1417726cb94dbc83.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/libstd-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/libpanic_unwind-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/libunwind-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/librand-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/libcollections-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/librustc_unicode-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/liballoc-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/liballoc_system-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/liblibc-39b92f95.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.q8eAleDtOvmp/libcore-39b92f95.rlib" "-Wl,--no-whole-archive" "-l" "ruby" "-l" "util" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "util" "-shared" "-l" "compiler-rt"
remote:        note: /usr/bin/ld: cannot find -lruby
remote:        collect2: error: ld returned 1 exit status
remote:        
remote:        error: aborting due to previous error
remote:        error: Could not compile `string-utility-rb-native`.
remote:        
remote:        To learn more, run the command again with --verbose.
remote:        make: *** [all] Error 101
remote:        
remote:        make failed, exit code 2
remote:        
remote:        Gem files will remain installed in /tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/gems/string-utility-2.7.3 for inspection.
remote:        Results logged to /tmp/build_b49520d155f32d44a5ec710ef7b84985/vendor/bundle/ruby/2.3.0/extensions/x86_64-linux/2.3.0-static/string-utility-2.7.3/gem_make.out
remote:        An error occurred while installing string-utility (2.7.3), and Bundler cannot
remote:        continue.
remote:        Make sure that `gem install string-utility -v '2.7.3'` succeeds before bundling.
emk commented 8 years ago

We may need help from a Ruby buildpack expert here. I think rust actually links with GCC.

emk commented 8 years ago

@schneems Do you have any idea why rustc might not be able to find -lruby while the Ruby buildpack is running, and we're trying to compile a gem using Rust? It's using the system linker as far as I can tell from the logs.

I'm happy to provide you with commit access to this repository, by the way, to help with Ruby/Rust buildpack integration issues. I'd really love to get all these bugs sorted out so that we can write some Ruby+Rust tutorials!

schneems commented 8 years ago

@elifoster how are you setting up the app? What is the output of $ heroku buildpacks ? Do you have any config on the app? Is the gem built in rust string-utility ?

elifoster commented 8 years ago

What is the output of $ heroku buildpacks ?

=== rustrubytest Buildpack URLs
1. https://github.com/emk/heroku-buildpack-rust
2. heroku/ruby

Do you have any config on the app?

What do you mean?

Is the gem built in rust string-utility ?

Yes, string-utility is built in part with Rust, and that is the gem I am testing this with.

schneems commented 8 years ago

I tried installing that gem locally and can't. I've got cargo installed and on the path, is there anything else I need?

$ cargo -V
cargo 0.13.0-nightly (109cb7c 2016-08-19)
2.3.1  ~/documents/projects/tmp/heroku-ruby-rust-hello (master)
$ bundle install
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using rack 1.6.4
Using tilt 2.0.4
Using bundler 1.13.1
Using rack-protection 1.5.3
Using sinatra 1.4.7
Installing string-utility 2.7.3 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3/ext
/Users/richardschneeman/.rubies/ruby-2.3.1/bin/ruby -r ./siteconf20161003-88028-xqeqig.rb extconf.rb
cargo 0.13.0-nightly (109cb7c 2016-08-19)
rustc 1.12.0 (3191fbae9 2016-09-23)

current directory: /Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3/ext
make "DESTDIR=" clean
rm -rf target

current directory: /Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3/ext
make "DESTDIR="
cargo build --release
    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling ruby-sys v0.2.13
   Compiling libc v0.2.16
   Compiling ruru v0.7.8
   Compiling string-utility-rb-native v1.0.0 (file:///Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3/ext)
warning: unused import, #[warn(unused_imports)] on by default
 --> src/lib.rs:4:12
  |
4 | use ruru::{AnyObject, Class, RString, VM};
  |            ^^^^^^^^^

warning: unused import, #[warn(unused_imports)] on by default
 --> src/lib.rs:4:39
  |
4 | use ruru::{AnyObject, Class, RString, VM};
  |                                       ^^

warning: unused import, #[warn(unused_imports)] on by default
 --> src/lib.rs:5:19
  |
5 | use ruru::types::{Argc, Value};
  |                   ^^^^

warning: unused import, #[warn(unused_imports)] on by default
 --> src/lib.rs:5:25
  |
5 | use ruru::types::{Argc, Value};
  |                         ^^^^^

warning: unused import, #[warn(unused_imports)] on by default
 --> src/lib.rs:6:5
  |
6 | use ruru::traits::Object;
  |     ^^^^^^^^^^^^^^^^^^^^

error: linking with `cc` failed: exit code: 1
  |
= note: "cc" "-m64" "-L" "/Users/richardschneeman/.multirust/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib"
"/Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3/ext/target/release/deps/stringutility.0.o" "-o"
"/Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3/ext/target/release/deps/libstringutility.dylib"
"/Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3/ext/target/release/deps/stringutility.metadata.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L"
"/Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3/ext/target/release/deps" "-L" "/Users/richardschneeman/.rubies/ruby-2.3.1/lib" "-L"
"/Users/richardschneeman/.multirust/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/libruru-35c12ee1cec9de02.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/libruby_sys-a3c0096998cfe6fb.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/liblibc-1417726cb94dbc83.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/libstd-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/libpanic_unwind-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/libunwind-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/librand-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/libcollections-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/librustc_unicode-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/liballoc-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/liballoc_system-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/liblibc-40393716.rlib"
"-Wl,-force_load,/var/folders/ss/lf77xfjd1g7ftznjmk3jzw1w0000gn/T/rustc.Pjd6qcs77GZk/libcore-40393716.rlib" "-l" "ruby" "-l" "System" "-l" "pthread" "-l" "c" "-l" "m" "-dynamiclib"
"-Wl,-dylib" "-l" "compiler-rt"
  = note: Undefined symbols for architecture x86_64:
  "_rb_sym2id", referenced from:
      ruru::class::symbol::Symbol::to_string::h9c3f5761208977c6 in libruru-35c12ee1cec9de02.rlib(ruru-35c12ee1cec9de02.0.o)
  "_rb_id2sym", referenced from:
      ruru::class::symbol::Symbol::new::h605e3540c8e2c62d in libruru-35c12ee1cec9de02.rlib(ruru-35c12ee1cec9de02.0.o)
  "_rb_funcallv", referenced from:
      ruru::binding::util::call_method::h8a74c812b5d32a7e in libruru-35c12ee1cec9de02.rlib(ruru-35c12ee1cec9de02.0.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error

error: Could not compile `string-utility-rb-native`.

To learn more, run the command again with --verbose.
make: *** [all] Error 101

make failed, exit code 2

Gem files will remain installed in /Users/richardschneeman/.gem/ruby/2.3.1/gems/string-utility-2.7.3 for inspection.
Results logged to /Users/richardschneeman/.gem/ruby/2.3.1/extensions/x86_64-darwin-14/2.3.0-static/string-utility-2.7.3/gem_make.out

An error occurred while installing string-utility (2.7.3), and Bundler cannot continue.
Make sure that `gem install string-utility -v '2.7.3'` succeeds before bundling
elifoster commented 8 years ago

No all you should need is rust/cargo and ruby. What OS are you on/is Heroku using?

schneems commented 8 years ago

I'm on a mac. Heroku is on ubuntu 14.04

$ heroku run bash
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:    14.04
Codename:   trusty
elifoster commented 8 years ago

Strange. I just uninstalled and reinstalled string-utility and it was fine. Also on OS X. It looks like that's an issue with ruru itself and not string-utility. Hrm.

schneems commented 8 years ago

Anything else I should be trying? Any other commands you want me to run to make sure my rust install is valid?

elifoster commented 8 years ago

Honestly I'm not sure. I just updated Cargo and Rust (I was a stable version behind, now on the same versions as you) and it still works fine for me. Are you/is Heroku using rbenv? This looks similar to d-unseductable/ruru#1 and d-unseductable/ruru#11

@d-unseductable perhaps you might have some thoughts regarding this?

d-unsed commented 8 years ago

It seems that the MRI which is used in ruby buildpack does not have a shared library. The library is linked to the gem after compilation (-lruby).

Shared library can be built while installing MRI using --enable-shared flag, for example:

If I understand correctly from the buildpack source code, ruby is downloaded from an S3 bucket. In this case, I see two options:

schneems commented 8 years ago

We are looking into maybe recompiling them all, no guarantees though.

On the other hand, do you know why it's needed in this case? Hundreds (or thousands?) of c-extensions on Rubygems are able to compile against Heroku's ruby versions without this requirement.

emk commented 8 years ago

I think the solution is to fix the build.rs for the Rust Ruby bindings to force static linking, maybe using a cargo feature. Shared libs are a pain in Heroku, and Rust likes static C libs well enough.

schneems commented 8 years ago

Want to add @hone onto this conversation.

d-unsed commented 8 years ago

@emk @schneems unfortunately, I could not manage to make it work with static library due to cargo limitations. It does not allow to pass all the required arguments to linker.

I will spend more time on it next week.

emk commented 5 years ago

Did anybody ever find a solution to building Ruby extensions written in Rust? I'm happy to devote some more time to this, but the underlying technologies have all changed significantly since this issue was opened, and I'm not sure where to start.

emk commented 4 years ago

This is old and nobody seems to working on it recently, so I'm going to close the issue. Thank you to everybody!