introduce some support for native gems

flavorjones commented 3 years ago

Native gems were briefly discussed at #8 and I was able to do a little bit of work to add support for native (precompiled) gems.

To try this out, you'll need Docker installed on your system. Then:

bundle update to install rake-compiler-dock
bundle exec rake clean
bundle exec rake gem:x86_64-linux to build that platform's native gem

I've also added x86-linux to this commit; but not Darwin because I had trouble getting libmagic to cross-compile (we can look into that later).

The packaged gem will the contain a lib directory that looks like:

lib/magic/2.5/magic.so
lib/magic/2.6/magic.so
lib/magic/2.7/magic.so
lib/magic/3.0/magic.so

Each of these is a dynamic shared library that contains a statically-linked version of libmagic, with the C extension compiled against that version of Ruby.

Let me know what you think, and I hope this is helpful.

stanhu commented 3 years ago

@flavorjones Thanks, this is awesome! I do see these warnings at the beginning of the build:

no configuration section for specified version of Ruby (rbconfig-x86-linux-2.5.0)
no configuration section for specified version of Ruby (rbconfig-x86-linux-2.6.0)
no configuration section for specified version of Ruby (rbconfig-x86-linux-2.7.0)
no configuration section for specified version of Ruby (rbconfig-x86-linux-3.0.0)

Is that expected?

After installing the gem, I also got this:

irb(main):001:0> require 'magic'
Traceback (most recent call last):
       10: from /Users/stanhu/.rbenv/versions/2.7.2/bin/irb:23:in `<main>'
        9: from /Users/stanhu/.rbenv/versions/2.7.2/bin/irb:23:in `load'
        8: from /Users/stanhu/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
        7: from (irb):1
        6: from /Users/stanhu/.rbenv/versions/2.7.2/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:156:in `require'
        5: from /Users/stanhu/.rbenv/versions/2.7.2/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:168:in `rescue in require'
        4: from /Users/stanhu/.rbenv/versions/2.7.2/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:168:in `require'
        3: from /Users/stanhu/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/ruby-magic-0.4.0-x86_64-linux/lib/magic.rb:3:in `<top (required)>'
        2: from /Users/stanhu/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/ruby-magic-0.4.0-x86_64-linux/lib/magic.rb:7:in `rescue in <top (required)>'
        1: from /Users/stanhu/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/ruby-magic-0.4.0-x86_64-linux/lib/magic.rb:7:in `require_relative'
LoadError (cannot load such file -- /Users/stanhu/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/ruby-magic-0.4.0-x86_64-linux/lib/magic/magic)

Seems we are missing the share files that are normally installed with the libmagic compilation.

flavorjones commented 3 years ago

no configuration section for specified version of Ruby

Yes, these are emitted by rake-compiler to warn about the platform/ruby-version matrix elements that that specific container can't build. Safe to ignore.

I can't reproduce the error you're seeing. You may want to uninstall any versions of ruby-magic that you already have installed, and then try to install the generate gem file again. Here's what I see (on x86_64):

$ gem install --local pkg/ruby-magic-0.4.0-x86_64-linux.gem 
Thank you for installing!
Successfully installed ruby-magic-0.4.0-x86_64-linux
1 gem installed

$ irb -rmagic
2.7.2 :001 > Magic.new.file("/bin/ls")
 => "ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=2f15ad836be3339dec0e2e6a3c637e08e48aacbd, for GNU/Linux 3.2.0, stripped" 
2.7.2 :002 >

stanhu commented 3 years ago

I noticed that paths is set to the local build directory instead of the installed path:

irb(main):001:0> require 'magic'
=> true
irb(main):002:0> Magic.new
=> #<Magic:0x000055b4b48a42d8 @flags=0, @paths=["/home/stanhu/ruby-magic/ports/x86_64-redhat-linux/libmagic/5.39/share/misc/magic"]>

If you run bundle exec rake clobber to clean up ports/archive, you see:

irb(main):001:0> require 'magic'
=> true
irb(main):002:0> Magic.new
Traceback (most recent call last):
        6: from /home/stanhu/.rbenv/versions/2.7.3/bin/irb:23:in `<main>'
        5: from /home/stanhu/.rbenv/versions/2.7.3/bin/irb:23:in `load'
        4: from /home/stanhu/.rbenv/versions/2.7.3/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
        3: from (irb):2
        2: from (irb):2:in `new'
        1: from (irb):2:in `initialize'
Magic::MagicError (could not find any valid magic files!)

We need the .gem to include the magic.mgc file inside the share path.

kwilczynski commented 3 years ago

Hi @stanhu,

I noticed that paths is set to the local build directory instead of the installed path:
irb(main):001:0> require 'magic'
=> true
irb(main):002:0> Magic.new
=> #<Magic:0x000055b4b48a42d8 @flags=0, @paths=["/home/stanhu/ruby-magic/ports/x86_64-redhat-linux/libmagic/5.39/share/misc/magic"]>
[...]

I have on my "to do" list an item which is about looking into changing the location of the compiled Magic database, so that we would move it perhaps into the lib/magic, but this requires a small patch most likely, as I don't think there is a way to influence location of where the database will be placed other than setting the prefix (which would make sense if we were to then follow with make install or make DESTDIR="/some/path/somewhere" install), which we aren't really doing in this case - also it does not quite solve the problem. :)

Krzysztof

flavorjones commented 3 years ago

Ah, what you're both saying makes sense to me now.

Nokogiri copies some "extra" files from libxml2 (the C header files, specifically) into the gem as well. I imagine ruby-magic could do this pretty easily, too, so long as we can resolve the path dynamically (to wherever it's installed). I'll poke around a bit this afternoon.

flavorjones commented 3 years ago

I've added some commits that will:

package the magic.mgc file in native gems at ext/magic/share/magic.mgc
use that database file if it exists (otherwise fall back to the current behavior)
remove patches, tarball, and the mini_portile2 dependency from native gems
add some test coverage for Magic#paths and Magic#load when the MAGIC env var is set

Let me know what you think!

flavorjones commented 3 years ago

Just to clarify what happens here ...

If the ruby platform gem is installed, and libmagic is compiled at install time, then Magic#paths will be the default compiled into libmagic, something like:

/path/to/gems/ruby-magic-0.4.0/ports/x86_64-pc-linux-gnu/libmagic/5.39/share/misc/magic

If the native platform gem is installed, then Magic#paths will be overridden, and will be the file we install, something like:

/path/to/gems/ruby-magic-0.4.0-x86_64-linux/ext/magic/share/magic.mgc

flavorjones commented 3 years ago

@kwilczynski reported an issue with the current packaging at https://github.com/kwilczynski/ruby-magic/pull/16#issuecomment-817890141 which we should fix.

stanhu commented 3 years ago

@flavorjones Looks good, thanks!

kwilczynski commented 3 years ago

Hi @flavorjones, thank you for working on this! Looks amazing!

I am seeing a similar problem to the one described by @stanhu in our CI, as per:

Not sure yet where the problem might be. I will look later.

Krzysztof

kwilczynski / ruby-magic

introduce some support for native gems #17