Windows problem (ffi-based ruby-vips)

[kleisauke]

In order to keep https://github.com/jcupitt/ruby-vips/issues/115 clean, I decided to open a new issue for Windows support.

Install instructions for Windows x64:

• Install Ruby using the RubyInstaller.
• Set the RUBY_DLL_PATH environment variable to libvips bin directory.
• Clone the FFI branch with git clone -b ffi-experiment https://github.com/jcupitt/ruby-vips.git
• Install ruby-vips:

  cd ruby-vips
  gem install bundler
  bundle install
  bundle exec rake install

• Apply this patch (my attempt to load the DLL files).
• Run a example:

  cd example
  trim8.rb image.png image2.png

You'll see this error:

 (LoadError)4/lib/ruby/gems/2.4.0/gems/ffi-1.9.18-x64-mingw32/lib/ffi/library.rb:147:in `block in ffi_lib': Could not open library 'libvips-42.dll': The specified procedure could not be found.
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ffi-1.9.18-x64-mingw32/lib/ffi/library.rb:100:in `map'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ffi-1.9.18-x64-mingw32/lib/ffi/library.rb:100:in `ffi_lib'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips.rb:356:in `<module:Vips>'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips.rb:353:in `<top (required)>'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `require'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `rescue in require'
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:40:in `require'
        from C:/ruby-vips/example/trim8.rb:10:in `<main>'

Tested on 2 different Windows 10 64-bits computer's, unfortunately the error persists. I've opened libvips-42.dll in Dependency Walker to see if I missed some dependencies, but all seems to be ok.

[kleisauke]

Update: It seems that the The specified procedure could not be found error only happens with Ruby 2.4.1-2 (x64) which is using the MSYS2 toolkit.

When I tried with Ruby 2.3.3 (x64) (from RubyInstaller, which is using the MSYS/MinGW based toolkit), libvips-42.dll is correctly loaded. I only see these GObject warnings now:

(trim8.rb:1892): GLib-GObject-WARNING **: gtype.c:4265: type id '0' is invalid

(trim8.rb:1892): GLib-GObject-WARNING **: can't peek value table for type '<invalid>' which is not currently referenced

(trim8.rb:1892): GLib-GObject-WARNING **: gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation
C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/ruby-vips-2.0.0/lib/vips/gvalue.rb:136:in `set': unimplemented gtype for set: 0 (Vips::Error)
        from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/ruby-vips-2.0.0/lib/vips/object.rb:129:in `set'
        from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:153:in `set'
        from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:312:in `block in call'
        from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:307:in `each_index'
        from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:307:in `call'
        from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/ruby-vips-2.0.0/lib/vips/image.rb:237:in `new_from_file'
        from C:/ruby-vips/example/trim8.rb:12:in `<main>'

New install instructions for Windows x64:

• Install Ruby 2.3.3 (x64) using the RubyInstaller.
• Install the DevKit:
  • Extract it to C:\Ruby23-x64\DevKit
  • Run ruby dk.rb init followed by ruby dk.rb install
• Add the libvips bin directory to the PATH environment variable.
• Clone the FFI branch with git clone -b ffi-experiment https://github.com/jcupitt/ruby-vips.git
• Install ruby-vips:

  cd ruby-vips
  gem install bundler
  bundle install
  bundle exec rake install

• Apply this patch (my attempt to load the DLL files).
• Run a example:

  cd example
  trim8.rb image.png image2.png

I'll now try to fix those GObject warnings and investigate why it was failing on Ruby 2.4.1-2 (x64).

[kleisauke]

I could reproduce The specified procedure could not be found error with Python. I've installed Python with Pacman (package management system) which ships in MSYS2:

Steps to reproduce this error:

• Install Python 3

  pacman -S mingw-w64-x86_64-python3

• Add C:\msys64\mingw64\bin to the PATH environment variable.

• Try to import the libvips-42.dll

  python3 -c "from ctypes import cdll; cdll.LoadLibrary(\"C:\\vips-8.5.6\\bin\\libvips-42.dll\")"

You'll see these errors: (The procedure entry point libintl_printf could not be located in the dynamic link library C:\vips-8.5.6\bin\libexif-12.dll.) (The procedure entry point libintl_printf could not be located in the dynamic link library C:\vips-8.5.6\bin\libvips-42.dll.)

And this in the console:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\msys64\mingw64\lib\python3.6\ctypes\__init__.py", line 426, in LoadLibrary
    return self._dlltype(name)
  File "C:\msys64\mingw64\lib\python3.6\ctypes\__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 127] The specified procedure could not be found

https://github.com/jcupitt/libvips/issues/500 seems to be related, not sure if this patch was landed in libvips 8.5.6.

[kleisauke]

I found the underlying problem, libintl-8.dll is getting included from the Ruby builtin DLL path (C:\Ruby24-x64\bin\ruby_builtin_dlls) instead than from the libvips bin directory (C:\vips-8.5.6\bin).

There's no libintl_printf in libintl 0.19.8.1, see:

In libintl 0.18.1 (included by libexif 0.6.21), there's a libintl_printf:

So after copying (and replacing) C:\vips-8.5.6\bin\libintl-8.dll to C:\Ruby24-x64\bin\ruby_builtin_dlls\libintl-8.dll the The specified procedure could not be found error is gone.

I now see:

(trim8.rb:860): GLib-GObject-WARNING **: gtype.c:4265: type id '0' is invalid

(trim8.rb:860): GLib-GObject-WARNING **: can't peek value table for type '<invalid>' which is not currently referenced

(trim8.rb:860): GLib-GObject-WARNING **: gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation
C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/gvalue.rb:136:in `set': unimplemented gtype for set: 0 (Vips::Error)
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/object.rb:129:in `set'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:153:in `set'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:312:in `block in call'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:307:in `each_index'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:307:in `call'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/image.rb:237:in `new_from_file'
        from trim8.rb:12:in `<main>'

Replacing a DLL feels hackish, is there another way? Maybe we could give the libvips bin directory a higher priority or we could exclude the Ruby builtin DLL path.

[jcupitt]

Hello @kleisauke I'm looking at this now. I've added your DLL load patch and I'll see if I can do anything about the libvips libintl.

I wonder why pyvips on Windows didn't have this problem? I guess they are not using libintl themselves.

[jcupitt]

The libvips binary is using this libintl:

ftp://ftp.gnome.org/pub/GNOME/binaries/win64/dependencies/gettext-runtime-dev_0.18.1.1-2_win64.zip

which is from 2010, though the latest that Gnome have.

I'll see if I can find out what RubyInstaller is using.

[jcupitt]

Looks like it's 0.19.8.1, see:

https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-gettext

I'll try to update the libvips one.

[kleisauke]

I think pyvips on Windows may have this problem if Python is installed with Pacman (which ships in MSYS2). I didn't check if it's broken for MSYS2 when making pyvips compatible for Windows (Python on my Windows PC is usually installed).

About the DLL load patch: I think it's a "dirty hack", obviously it would be nicer to split the GLib module into a GLib and GObject module and make the g_malloc, g_free call to the GLib module and the g_object_ref call to the GObject module.

The GLib module should than include the libglib-2.0-0.dll DLL for Windows and libgobject-2.0.so for Linux. Whereas the GObject module should include the libgobject-2.0-0.dll DLL for Windows and libgobject-2.0.so for Linux (maybe the GObject can inheritance from the GLib module if on Linux). I tried this with Ruby last week, but wasn't able to complete this (I don't have much experience with Ruby).

[jcupitt]

I agree, that would be prettier. I'll see if I can tweak it a bit.

It seems to be building with latest gettext, I'll do a test upload somewhere.

[kleisauke]

Is is possible to update libintl without breaking the libexif dependency (version 0.6.21)? libexif is causing these issues because it have a dependency to libintl 0.18.1.

By the way, libexif seems not to be updated since 2013 (which is 4 years ago), is there a alternative for libvips?

[jcupitt]

Yes, I'm rebuilding libexif too, so it should be OK, I think.

There's not a good alternative to libexif, unfortunately :-( there's talk of a new version incorporating all the accumulated fixes, but it hasn't happened yet.

We did look at switching to exiv2, but it's not possible for a variety of reasons, see https://github.com/jcupitt/libvips/issues/453.

[kleisauke]

I've just tested out https://github.com/jcupitt/ruby-vips/commit/17d42c9f5c497186210fe7b9d0148f54da220ff1.

With typedef :ulong, :GType here I still get this error:

(trim8.rb:6948): GLib-GObject-WARNING **: gtype.c:4265: type id '0' is invalid

(trim8.rb:6948): GLib-GObject-WARNING **: can't peek value table for type '<invalid>' which is not currently referenced

(trim8.rb:6948): GLib-GObject-WARNING **: gvalue.c:188: cannot initialize GValue with type '(NULL)', this type has no GTypeValueTable implementation
C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/gvalue.rb:136:in `set': unimplemented gtype for set: 0 (Vips::Error)
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/object.rb:129:in `set'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:153:in `set'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:312:in `block in call'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:307:in `each_index'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/operation.rb:307:in `call'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/ruby-vips-2.0.0/lib/vips/image.rb:250:in `new_from_file'
        from C:/ruby-vips/example/trim8.rb:12:in `<main>'

Changing it to typedef :uint64, :GType, then everything seems to work fine on Windows x64 (running trim8.rb image.png image2.png will successfully trim the image). :tada:

I didn't test it out on a 32-bit Windows PC, but I think it will need typedef :uint32, :GType (just like lua-vips and pyvips).

[jcupitt]

Great! \o/ I'll add the gtype change.

What about the libintl problem, is this with the DLL copy hack?

[kleisauke]

This is with the DLL copy hack. To check for 32-bit you could use FFI::Platform::ARCH == 'i386' and for 64-bit FFI::Platform::ARCH == 'x86_64'.

[jcupitt]

I found FFI::Platform::ADDRESS_SIZE which is 64 or 32, and should work on ppc and arm.

[kleisauke]

Ah, I didn't find that. That will do the job (and it's indeed better)! 👍

[jcupitt]

OK, here's a test build with current gettext, it seems to work:

http://www.rollthepotato.net/~john/vips-dev-w64-web-8.5.7.zip

and I pushed that gtype change. Would you be able to test again, Kleis?

[kleisauke]

I've tested https://github.com/jcupitt/ruby-vips/commit/cdb2061d578c464466b757f8808c5a1e3c03f276 with the provided libvips test build and it seems to be working! :tada:

First I completely removed/uninstalled Ruby and MSYS2 to ensure that the DLL copy hack(s) are gone. Then I re-installed Ruby (using the RubyInstaller which includes the MSYS2 toolkit) and set the RUBY_DLL_PATH environment variable to the libvips 8.5.7 bin directory.

After Ruby was installed, I tested the trim8.rb example. It trimmed the image fast and correctly.

The next thing I had tested were the rspec tests (with bundle exec rake spec). Many permission denied errors while deleting the temporary files appeared (déjà vu from pyvips), see:

Failure/Error: FileUtils.rm Dir[tmp.join('*.*')]

      Errno::EACCES:
        Permission denied @ unlink_internal - C:/ruby-vips/spec/tmp/working/x.v

After applying this patch:

--- C:/ruby-vips/spec/spec_helper.rb    Tue Aug 22 13:32:06 2017
+++ C:/ruby-vips/spec/spec_helper.rb    Tue Aug 22 13:32:29 2017
@@ -23,7 +23,7 @@

     module Helpers
         def reset_working!
-            FileUtils.rm Dir[tmp.join('*.*')]
+            FileUtils.rm Dir[tmp.join('*.*')], :force => true
             FileUtils.mkdir_p(tmp)
         end
     end

The rspec tests were successfully completed:

......................................................

Finished in 0.54074 seconds (files took 0.18814 seconds to load)
54 examples, 0 failures

I think Windows support is now done for the ffi-based ruby-vips, you may close this issue. :smile:

[jcupitt]

omg

/me falls off his chair

[jcupitt]

Good job Kleis! I'll apply that fix (I'd never heard of :force) and I think we can release 2.0.