Closed yorickpeterse closed 3 years ago
lupine
commented
3 years ago I had this, and found it was caused by my ruby (installed via ruby-install) not being compiled with --enable-shared
Running:
ruby-install ruby 2.7.2 -- --enable-sharedwas enough to generate the required libruby.so and fix linking:
$ ls ~/.rubies/ruby-2.7.2/lib
libruby.so libruby.so.2.7 libruby.so.2.7.2 libruby-static.a ruby(previously, I only had the libruby-static.a)
yorickpeterse
commented
3 years ago Seems I have the same problem. This does make me wonder if this should be necessary in the first place. I think this is the first C extension I've come across that requires a shared Ruby library to be available.
yorickpeterse
commented
3 years ago Arch Linux at least compiles with this option by default: https://github.com/archlinux/svntogit-packages/blob/packages/ruby/trunk/PKGBUILD. I guess it's not too weird to require that, though I'm still curious as to why this Gem needs a shared library.
kwilczynski
commented
3 years ago Hi @YorickPeterse, apologies for issues!
I need to have a closer look. I am able to build it on macOS and on Linux machine running Debian. It also seems fine when building on Ubuntu provided by Travis CI.
This is surprising, as I would expect issues with libmagic in terms of building and linking against, rather than with Ruby.
I personally use rbenv with ruby-install to manage newer (or more modern) versions of Ruby, but since you are using Arch, you would be using something recent.
Having said that, this might be a side-effect of building libmagic (which is now a dependency that we vendor since the https://github.com/kwilczynski/ruby-magic/pull/3) as a shared object, and I wonder if mkmf would somehow decide to set everything up so that I requires a shared Ruby too.
If you have libmagic installed somewhat, then you could try to remove the part of extconf.rb that now attempts to build libmagic, and just rely on mkmf detecting dependencies from your environment.
Just an idea to exclude the recent changes as a possible culprit.
Krzysztof
erikh
commented
3 years ago [37] erikh@islay ~% gem install ruby-magic [21:39]
Fetching ruby-magic-0.3.1.gem
WARNING: You don't have /home/erikh/.gem/ruby/2.7.0/bin in your PATH,
gem executables will not run.
Building native extensions. This could take a while...
Thank you for installing!
Successfully installed ruby-magic-0.3.1
1 gem installed
[38] erikh@islay ~% lsb_release -rt [21:39]
lsb_release: invalid option -- 't'
[fail:2][39] erikh@islay ~% lsb_release -r [21:39]
Release: 21.0
[40] erikh@islay ~% lsb_release -a [21:39]
LSB Version: n/a
Distributor ID: ManjaroLinux
Description: Manjaro Linux
Release: 21.0
Codename: Ornara
[41] erikh@islay ~% ruby -v [21:39]
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]Seems to work here.
stanhu
commented
3 years ago ruby-magic doesn't need a shared library, but getting the Ruby C extension to link against a static libmagic requires getting all the linker flags right via pkg-config. I attempted to do this in https://github.com/kwilczynski/ruby-magic/pull/5, but struggled to get it to work. I may do a deeper dive to figure this out.
stanhu
commented
3 years ago libmagic, but it still appears to build a magic.so for me. I've noticed that even Nokogiri builds a .so. Ruby, at least on my system.Ruby does appear to have support for compiling a gem as a static library: https://github.com/ruby/ruby/blob/cc8e966e3c8b44dd946c4c50cc8d172e1d45e625/lib/mkmf.rb#L2495-L2497. I'm not sure the conditions this is enabled. I've noticed adding $static = true in extconf.rb seems to append that Makefile blurb, but I'm not sure yet under what conditions the static library is built.
kwilczynski
commented
3 years ago Hi @YorickPeterse,
With https://github.com/kwilczynski/ruby-magic/pull/5 now merged (thank you again @stanhu for all the work), do you still have issues building and/or installing? If you have a moment, would you be able to check again?
Thank you in advance!
Krzysztof
yorickpeterse
commented
3 years ago @kwilczynski Using the latest master commit, it seems a Ruby installation with support for shared libraries is still required. I think this is mostly fine though, as this option seems to be widely enabled by default anyway.
stanhu
commented
3 years ago It looks like docker-gitlab also needed to add --enabled-shared: https://github.com/genshen/docker-gitlab/commit/c187b46525244126bb55c4ef94e1975e7278f9be. I wonder why.
stanhu
commented
3 years ago I replicated the problem by compiling a Ruby interpreter without --enabled-shared.
I'm not sure why the linker is requiring these rb_ symbols to be statically linked. With nokogiri, the .o looks to be in a similar state:
root@ba6c87769e16:/usr/local/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.2/ext/nokogiri# objdump -t nokogiri.o | grep rb_ | grep UND
0000000000000000 *UND* 0000000000000000 rb_funcallv
0000000000000000 *UND* 0000000000000000 rb_rescue
0000000000000000 *UND* 0000000000000000 rb_string_value_ptr
0000000000000000 *UND* 0000000000000000 rb_str_new
0000000000000000 *UND* 0000000000000000 rb_num2int
0000000000000000 *UND* 0000000000000000 rb_fix2int
0000000000000000 *UND* 0000000000000000 rb_define_module
0000000000000000 *UND* 0000000000000000 rb_define_module_under
0000000000000000 *UND* 0000000000000000 rb_utf8_encoding
0000000000000000 *UND* 0000000000000000 rb_external_str_new_with_enc
0000000000000000 *UND* 0000000000000000 rb_const_set
0000000000000000 *UND* 0000000000000000 rb_str_split
0000000000000000 *UND* 0000000000000000 rb_eStandardError
0000000000000000 *UND* 0000000000000000 rb_define_class_under
0000000000000000 *UND* 0000000000000000 rb_intern2The compilation step looks similar:
gcc -shared -o nokogiri.so html_document.o html_element_description.o html_entity_lookup.o html_sax_parser_context.o html_sax_push_parser.o libxml2_backwards_compat.o nokogiri.o test_global_handlers.o xml_
attr.o xml_attribute_decl.o xml_cdata.o xml_comment.o xml_document.o xml_document_fragment.o xml_dtd.o xml_element_content.o xml_element_decl.o xml_encoding_handler.o xml_entity_decl.o xml_entity_reference
.o xml_namespace.o xml_node.o xml_node_set.o xml_processing_instruction.o xml_reader.o xml_relax_ng.o xml_sax_parser.o xml_sax_parser_context.o xml_sax_push_parser.o xml_schema.o xml_syntax_error.o xml_tex
t.o xml_xpath_context.o xslt_stylesheet.o -L. -L/usr/local/lib -Wl,-rpath,/usr/local/lib -L/usr/local/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.2/ports/x86_64-pc-linux-gnu/libxml2/2.9.10/lib -Wl,-rpath,/usr/l
ocal/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.2/ports/x86_64-pc-linux-gnu/libxml2/2.9.10/lib -L/usr/local/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.2/ports/x86_64-pc-linux-gnu/libxslt/1.1.34/lib -Wl,-rpath,/usr
/local/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.2/ports/x86_64-pc-linux-gnu/libxslt/1.1.34/lib -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--compress-debug-sections=zlib /usr/local/lib/
ruby/gems/2.7.0/gems/nokogiri-1.11.2/ports/x86_64-pc-linux-gnu/libxslt/1.1.34/lib/libexslt.a -lm -ldl -lz /usr/local/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.2/ports/x86_64-pc-linux-gnu/libxml2/2.9.10/lib/li
bxml2.a /usr/local/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.2/ports/x86_64-pc-linux-gnu/libxslt/1.1.34/lib/libxslt.a -ldl -lm -lz /usr/local/lib/ruby/gems/2.7.0/gems/nokogiri-1.11.2/ports/x86_64-pc-linux-gnu
/libxml2/2.9.10/lib/libxml2.a -lm -lcruby-magic has a few more items:
root@ba6c87769e16:/tmp/ruby-magic# objdump -t ruby-magic.o | grep rb_ | grep UND
0000000000000000 *UND* 0000000000000000 rb_check_type
0000000000000000 *UND* 0000000000000000 rb_exc_new_cstr
0000000000000000 *UND* 0000000000000000 rb_funcallv
0000000000000000 *UND* 0000000000000000 rb_intern2
0000000000000000 *UND* 0000000000000000 rb_thread_call_without_gvl
0000000000000000 *UND* 0000000000000000 rb_protect
0000000000000000 *UND* 0000000000000000 rb_iv_set
0000000000000000 *UND* 0000000000000000 rb_jump_tag
0000000000000000 *UND* 0000000000000000 rb_data_object_wrap
0000000000000000 *UND* 0000000000000000 rb_exc_raise
0000000000000000 *UND* 0000000000000000 rb_str_new_cstr
0000000000000000 *UND* 0000000000000000 rb_str_new_static
0000000000000000 *UND* 0000000000000000 rb_cInteger
0000000000000000 *UND* 0000000000000000 rb_class2name
0000000000000000 *UND* 0000000000000000 rb_obj_classname
0000000000000000 *UND* 0000000000000000 rb_eTypeError
0000000000000000 *UND* 0000000000000000 rb_raise
0000000000000000 *UND* 0000000000000000 rb_gc_mark
0000000000000000 *UND* 0000000000000000 rb_ivar_get
0000000000000000 *UND* 0000000000000000 rb_ensure
0000000000000000 *UND* 0000000000000000 rb_obj_is_kind_of
0000000000000000 *UND* 0000000000000000 rb_num2int
0000000000000000 *UND* 0000000000000000 rb_string_value_cstr
0000000000000000 *UND* 0000000000000000 rb_fix2int
0000000000000000 *UND* 0000000000000000 rb_string_value
0000000000000000 *UND* 0000000000000000 rb_respond_to
0000000000000000 *UND* 0000000000000000 rb_convert_type
0000000000000000 *UND* 0000000000000000 rb_io_taint_check
0000000000000000 *UND* 0000000000000000 rb_io_check_closed
0000000000000000 *UND* 0000000000000000 rb_eIOError
0000000000000000 *UND* 0000000000000000 rb_ary_entry
0000000000000000 *UND* 0000000000000000 rb_ary_new
0000000000000000 *UND* 0000000000000000 rb_ivar_set
0000000000000000 *UND* 0000000000000000 rb_warn
0000000000000000 *UND* 0000000000000000 rb_block_given_p
0000000000000000 *UND* 0000000000000000 rb_eval_string
0000000000000000 *UND* 0000000000000000 rb_cObject
0000000000000000 *UND* 0000000000000000 rb_const_get
0000000000000000 *UND* 0000000000000000 rb_class_new_instance
0000000000000000 *UND* 0000000000000000 rb_uint2big
0000000000000000 *UND* 0000000000000000 rb_num2ulong
0000000000000000 *UND* 0000000000000000 rb_eArgError
0000000000000000 *UND* 0000000000000000 rb_define_class
0000000000000000 *UND* 0000000000000000 rb_define_alloc_func
0000000000000000 *UND* 0000000000000000 rb_eStandardError
0000000000000000 *UND* 0000000000000000 rb_define_class_under
0000000000000000 *UND* 0000000000000000 rb_define_attr
0000000000000000 *UND* 0000000000000000 rb_alias
0000000000000000 *UND* 0000000000000000 rb_define_const
0000000000000000 w *UND* 0000000000000000 rb_define_method
0000000000000000 w *UND* 0000000000000000 rb_define_singleton_methodHere's the compile step failing in ruby-magic:
gcc -shared -o magic.so functions.o ruby-magic.o -L. -L/usr/local/lib -Wl,-rpath,/usr/local/lib -L/tmp/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib -Wl,-rpath,/tmp/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-rpath,/tmp/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib -Wl,--as-needed -Wl,--no-undefined -Wl,--exclude-libs,ALL -Wl,--compress-debug-sections=zlib -lmagic -lm -lc -L/tmp/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib -lmagic -lz /tmp/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib/libmagic.aAttempting to add -lruby-static also fails:
gcc -shared -o magic.so functions.o ruby-magic.o -L. -L/usr/local/lib -Wl,-rpath,/usr/local/lib -L/tmp/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib -Wl,-rpath,/tmp/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-rpath,/tmp/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib -Wl,--as-needed -Wl,--no-undefined -Wl,--exclude-libs,ALL -Wl,--compress-debug-sections=zlib -lmagic -lm -lc -lruby-static -fPIC -lm
/usr/bin/ld: /usr/local/lib/libruby-static.a(error.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(gc.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(hash.o): relocation R_X86_64_PC32 against symbol `environ@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(io.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(mjit.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(mjit_compile.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(thread.o): relocation R_X86_64_PC32 against symbol `__libc_stack_end@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(transient_heap.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(version.o): relocation R_X86_64_PC32 against symbol `stdout@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(vm.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(vm_backtrace.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(vm_dump.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(setproctitle.o): relocation R_X86_64_PC32 against symbol `environ@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(addr2line.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(compile.o): relocation R_X86_64_PC32 against symbol `stdout@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/local/lib/libruby-static.a(debug.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: nonrepresentable section on outputThe presence of -shared though suggests to me that we can't make a shared object, so I'm a little puzzled why nokogiri compiles without a hitch.
kwilczynski
commented
3 years ago Hi @stanhu,
The culprit here most likely will be the -Wl,--no-undefined linker option, as per the undefined symbols reported by objdump. This flag would cause linker to disallow any symbols that are unknown at linking time. It's meant to catch potential issues, but sometimes - like in this case too - can cause troubles.
We add it here:
Krzysztof
stanhu
commented
3 years ago Ah, thanks, I confirmed that fixes the problem. Can we just drop those lines?
kwilczynski
commented
3 years ago Hi @stanhu,
Ah, thanks, I confirmed that fixes the problem. Can we just drop those lines?
We certainly can. I would imagine all the symbols are there and things will resolve when dlopen() loads our shared object.
Krzysztof
kwilczynski
commented
3 years ago Hi @stanhu,
Thank you for digging into this problem and fixing it!
Krzysztof
This MR removed the following line:
On my Arch Linux installation this is now leading to the following compile error:
It seems that this change, for whatever reason, is resulting in the right headers not being found? I'm not sure yet what's going on exactly.