new ffi-based ruby-vips needs testing

jcupitt commented 7 years ago

There's a branch with a new version of ruby-vips, this time based on ffi rather than gobject-introspection:

https://github.com/jcupitt/ruby-vips/tree/ffi-experiment

Good things:

It passes the same test suite, so hopefully the API is exactly the same.
It should be much more portable: it just uses ffi, so there's no need to get gobject-introspection working. It should work with things like JRuby as well.
It's only 500 lines longer than the old gem.
It runs the test suite about about twice as fast as the old gem.
It should be more stable and more reliable.
Installation should be much simpler. You don't need to set up typelibs or anything like that.

Bad things:

It needs testing!
The docs need some updating still.

tomasc commented 7 years ago

@jcupitt will test it on staging env of one of my projects and report back.

jcupitt commented 7 years ago

OK, I think it's pretty much done. I've leaktested it and updated the docs, it all seems stable, as far as I can see, and all my old code runs OK.

It just needs some testing!

harukizaemon commented 7 years ago

OMG this is incredible! I've just been porting an app from Ruby to Java and was considering re-writing the VIPS pieces as a standalone C++ executable. You may have just saved me weeks of effort.

jcupitt commented 7 years ago

Oh, you mean with JRuby? It would be great if you could test it --- it should work, according to the ruby-ffi README, but I've not tried it.

jcupitt commented 7 years ago

It has keyword arg support now, so it's ruby 2.0 and later only, FYI.

tomasc commented 7 years ago

I had to remove the optimize_coding option (caused issues when saving JPG), but other than that it seems to work fine! I will keep it running on staging env of one of our websites for a bit to see if we encounter any problems.

jcupitt commented 7 years ago

That's great!

I think I found the optimize_coding problem, could you try again?

tomasc commented 7 years ago

Yes, seems to be resolved now!

kleisauke commented 7 years ago

On Windows it has some problems loading the gobject library. This was the first error:

 (LoadError)4/lib/ruby/gems/2.4.0/gems/ffi-1.9.18-x64-mingw32/lib/ffi/library.rb:147:in `block in ffi_lib': Could not open library 'gobject-2.0': The specified module could not be found.
.
Could not open library 'gobject-2.0.dll': The specified module could not be found.
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ffi-1.9.18-x64-mingw32/lib/ffi/library.rb:100:in `map'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ffi-1.9.18-x64-mingw32/lib/ffi/library.rb:100:in `ffi_lib'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips.rb:14:in `<module:GLib>'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips.rb:12:in `<top (required)>'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `require'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `rescue in require'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:40:in `require'
        from C:/ruby-vips/example/test.rb:1:in `<main>'

That error is fixable after applying this patch:

--- vips.rb 2017-07-22 12:31:47.502401900 +0200
+++ vips.rb 2017-07-22 12:33:14.694250100 +0200
@@ -11,7 +11,13 @@

 module GLib
     extend FFI::Library
-    ffi_lib 'gobject-2.0'
+    if FFI::Platform.windows?
+      libgobject = 'C:\vips-dev-w64-all-8.5.6\vips-dev-8.5\bin\libgobject-2.0-0.dll'
+    else
+      libgobject = 'gobject-2.0'
+    end
+    
+    ffi_lib libgobject

     # nil being the default
     glib_log_domain = nil

(I tried using libgobject = 'libgobject-2.0-0.dll' but that doesn't work, the PATH environment variable is correctly set)

When I applied the above patch, this was the next error:

C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ffi-1.9.18-x64-mingw32/lib/ffi/library.rb:275:in `attach_function': Function 'g_malloc' not found in [C:\vips-dev-w64-all-8.5.6\vips-dev-8.5\bin\libgobject-2.0-0.dll] (FFI::NotFoundError)
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips.rb:29:in `<module:GLib>'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips.rb:12:in `<top (required)>'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `require'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `rescue in require'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:40:in `require'
        from test.rb:1:in `<main>'

I'll try on Linux instead. 😄

jcupitt commented 7 years ago

Hello Kleis, that's very strange, it should search for DLLs in the current directory, and then along PATH.

What do you have it set to? If you have the directory C:/vips-dev-w64-all-8.5.6/vips-dev-8.5/bin at the front, you should be OK. Could it be a / and \ mixup? Or it interpreting \v as an escape character? And you either must or must not have a trailing / or \, I forget which.

It took a bit of fiddling, but I did get the lua-vips ffi binding working under Wine. The equivalent code is here:

https://github.com/jcupitt/lua-vips/blob/master/src/vips/gvalue.lua#L14

kleisauke commented 7 years ago

According to https://github.com/oneclick/rubyinstaller2/issues/4 (I'm using RubyInstaller) it searches a relative FFI load in C:\msys64\mingw64\bin (for x86 in C:\msys64\mingw32\bin). So after copying libgobject-2.0-0.dll and libglib-2.0-0.dll to C:\msys64\mingw64\bin and applying this patch:

--- vips.rb Sat Jul 22 12:31:47 2017
+++ vips.rb Sat Jul 22 14:47:10 2017
@@ -11,7 +11,12 @@

 module GLib
     extend FFI::Library
-    ffi_lib 'gobject-2.0'
+    if FFI::Platform.windows?
+      ffi_lib 'libgobject-2.0-0.dll'
+      ffi_lib 'libglib-2.0-0.dll'
+    else
+      ffi_lib 'gobject-2.0'
+    end

     # nil being the default
     glib_log_domain = nil

gobject and glib are successfully loaded.

Unfortunately, I now get a new error:

C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ffi-1.9.18-x64-mingw32/lib/ffi/library.rb:275:in `attach_function': Function 'g_type_name' not found in [libglib-2.0-0.dll] (FFI::NotFoundError)
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips.rb:37:in `<module:GLib>'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips.rb:12:in `<top (required)>'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `require'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `rescue in require'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:40:in `require'
        from test.rb:1:in `<main>'

jcupitt commented 7 years ago

Oh dear, so ruby-ffi does not use any search path for loading DLLs, just absolute and relative filenames. I wonder what their reasoning is for turning off the standard DLL search mechanism? Security I guess.

I suppose this means we must locate the directory containing libvips.dll and libgobject.dll ourselves somehow. How about just getenv("PATH") and looking along that?

jcupitt commented 7 years ago

Thinking about it some more, I suppose the problem is that PATH is set per user on Windows, but many programs might need different settings (if they need different versions of the same DLL). On other systems, PATH is generally per installed program (via wrapper scripts), but wrapper scripts don't really exist on Windows, so each program is forced to locate its own DLLs.

The solutions seem to be:

A new environment variable, perhaps VIPSHOME, which points to the vips binary directory.
A registry key pointing to the vips binary. This would make installing annoying.
The calling program gives ruby-vips the directory of the vips binary somehow.
When we package for Windows, we include the vips binary in the gem and locate it in there somehow.

Does anyone have any better ideas? This is painful :(

kleisauke commented 7 years ago

After some searching I found a solution. Setting the RUBY_DLL_PATH environment variable to C:\vips-dev-w64-all-8.5.6\vips-dev-8.5\bin fixes this. See: https://github.com/oneclick/rubyinstaller2/issues/51

tomasc commented 7 years ago

I have been using the ffi-experiment branch for several weeks now and am not seeing any issues.

jcupitt commented 7 years ago

Looks like it's done! Windows support is tested too, and it all seems to work. JRuby might be nice to test too, but that can wait.

I've pushed 2.0 to rubygems.

tomasc commented 7 years ago

fabulous, thank you @jcupitt!

libvips / ruby-vips

new ffi-based ruby-vips needs testing #115