Solution: OSX workaround for using pyvips under anaconda with homebrew vips

Hi John,

I've recently had to install libvips and pyvips on a new MacBook Pro for work, and I encountered an identical issue to #24 and #38 by trying to mix homebrew vips and anaconda python, resulting in all tests failing with loads of error messages from libglib-2.0 and libgobject-2.0. It was quite important for me to have pyvips running under a conda-based configuration, so I poked around and I think I've found an easy workaround.

I had previously installed vips via homebrew. When subsequently building pyvips, it was failing to build an API version despite pkg-config vips --libs printing the correct information. The API-compilation problem was revealed by running the below command; pkg-config didn't find libffi:

$ pkg-config --exists --print-errors vips
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libffi', required by 'gobject-2.0', not found

That problem was fixed with $ brew install libffi

followed by

$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

Once these changes were made, running python setup.py build in the root directory of the pyvips package actually compiled build/lib.macosx-10.7-x86_64-3.7/_libvips.abi3.so, but I was getting the same errors as #24 and #38 (GLib-CRITICAL **: g_datalist_id_set_data_full: assertion 'key_id > 0' failed, e.g.). I suspected a clash between libvips' linked libraries and the ones in _libvips.abi3.so, specifically conflicting versions of libglib-2.0 and libgobject-2.0:

~/src/pyvips/build/lib.macosx-10.7-x86_64-3.7$ otool _libvips.abi3.so -L
_libvips.abi3.so:
    /usr/local/opt/vips/lib/libvips.42.dylib (compatibility version 53.0.0, current version 53.0.0)
    @rpath/libgobject-2.0.0.dylib (compatibility version 5601.0.0, current version 5601.2.0)
    @rpath/libglib-2.0.0.dylib (compatibility version 5601.0.0, current version 5601.2.0)
    @rpath/libintl.8.dylib (compatibility version 10.0.0, current version 10.5.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)

Compare to:

$ otool -L /usr/local/opt/vips/lib/libvips.42.dylib
/usr/local/opt/vips/lib/libvips.42.dylib:
    /usr/local/opt/vips/lib/libvips.42.dylib (compatibility version 53.0.0, current version 53.0.0)
        ...
    /usr/local/opt/glib/lib/libgobject-2.0.0.dylib (compatibility version 6001.0.0, current version 6001.3.0)
    /usr/local/opt/glib/lib/libglib-2.0.0.dylib (compatibility version 6001.0.0, current version 6001.3.0)
        ...

I manually set the libraries in _libvips.abi3.so to point to the same ones as libvips by running the following:

$ install_name_tool -change "@rpath/libglib-2.0.0.dylib" "/usr/local/opt/glib/lib/libglib-2.0.0.dylib" _libvips.abi3.so
$ install_name_tool -change "@rpath/libgobject-2.0.0.dylib" "/usr/local/opt/glib/lib/libgobject-2.0.0.dylib" _libvips.abi3.so

After that, all tests passed, and running pip install . installed under anaconda and now everything works fine as far as I can tell (python setup.py pytest or tox -e py37-test pass all 28 tests and my own pyvips code seems to work fine so far).

I should caution that I've never used a Mac before, so when it comes to OSX I truly have no idea what I'm doing so there's probably a better, more correct way to do this (setting RPATH?). However, this worked for me after a day of frustration, so maybe this will help someone else avoid the same pain.

As a follow-up, the value of @rpath can be embedded inside the dynamic library, either during compilation or post facto with the install_name_tool command [1,2], so as an alternate to the above steps to hard-code the two library locations, the following also makes pyvips pass all tests:

~/src/pyvips$ install_name_tool -add_rpath /usr/local/opt/glib/lib/ build/lib.macosx-10.7-x86_64-3.7/_libvips.abi3.so

One can see the value of @rpath stored in the file (if any) by using otool -l and looking for LC_RPATH:

~/src/pyvips$ otool -l build/lib.macosx-10.7-x86_64-3.7/_libvips.abi3.so
build/lib.macosx-10.7-x86_64-3.7/_libvips.abi3.so:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           8    16       1808 0x00000085

...

Load command 15
          cmd LC_RPATH
      cmdsize 40
         path /usr/local/opt/glib/lib/ (offset 12)

On the compile side, I found that the value of rpath can be set as an extra linker flag in the ffibuilder, by adding this extra line in pyvips_build.py

ffibuilder.set_source("_libvips",
    r""" 
        #include <vips/vips.h>
    """, 
    **pkgconfig.parse('vips'),
    extra_link_args=['-Wl,-rpath,/usr/local/opt/glib/lib']) # manually added this

The above obviates the need to use install_name_tool at all. Determining under which circumstances the above line would need to be added is beyond the scope of my severely limited OSX expertise.

[1] https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172 [2] https://wincent.com/wiki/@executable_path,_@load_path_and_@rpath

Hi Rob,

I'm glad it's working for you, but I'm afraid this is not a good solution :( Anaconda has its own package system and its own set of libraries and mixing them like this is not supported. For example, what if another conda package pulls in the conda glib-2.0 and the version doesn't match the one in homebrew? Now if you import both packages, you'll get run-time link errors. It can get much worse than that: you can get silent data corruption, mysterious random threading crashes, ... argh!

The only way to really fix it is to make a proper conda package. It's a shame conda lives it it's own weird tiny incompatible world, but of course that's the strength of a cross-platform packaging system too. The other fix is to use homebrew Python.

Hi John,

I think I understand the general principle, but what I think I don't understand is how this interacts with the OSX ability to have a .dylib that hard-links to specific other dependencies.

This sounds plausible that libvips and pyvips can live in their own homebrew world, linking to libraries from homebrew, while other conda packages can have their libraries loaded from the conda world. This is what I was seeing before I fixed the problem, as observed with DYLD_PRINT_LIBRARIES=1 python setup.py test: specifically that libvips was loading libgobject-2.0.0.dylib from one place (/usr/local/opt/glib/lib) and that other conda things were loading it from another place (somewhere like /opt/anaconda3/lib/...). In other words, both libraries were simultaneously loaded and in action, but the problem was just that _libvips.abi3.so wasn't hard-wired to use the same one as /usr/local/opt/vips/lib/libvips.42.dylib, and was instead picking up the one from conda.

So, I reasoned that I could hard-wire _libvips.abi3.so to live entirely in the homebrew world with its libraries, and that this would not have any effect on anything in the conda world since I wasn't changing any equivalent of LD_LIBRARY_PATH, and the conda things already know to use libraries under /opt/anaconda3.

Is there a possibility that there is some kind of cross-contamination, given that, after the manual override, _libvips.abi3.so and libvips.42.dylib dynamically link entirely into the homebrew world and conda things dynamically link entirely into the conda world?

Thanks, and apologies for the newbie question.

Yes, macos has a two-level linking system, so you can in theory have g_type_ new() call two different things from two points in the same executable. But that's only the linker -- what about all the other things that make up a program, like environment variables, threads, per-thread storage, quarks, file handles, pointers, etc. etc., and how they behave when references to them pass from one part of the program to another. It'll perhaps work some of the time if you are lucky, I think is the best you can say.

The macos two level linking system was designed for their "framework" system, where libraries can be added and moved about as separate packages, and it works well for libraries which are designed with that in mind. But most standard *nix software like glib, was never designed to work like that, and it's at best an unsupported hack, unfortunately.

https://stackoverflow.com/questions/4618027/what-exactly-is-a-framework-in-mac-os-x-framework-folders

Perhaps I'm being too negative! Anyway, I think a proper conda package would be a better solution.

libvips / pyvips

Solution: OSX workaround for using pyvips under anaconda with homebrew vips #102