Version mismatch on OSX 10.8.4

[vhaasteren]

Hi nirinA,

I am starting to help Scott Ransom and others with their new pulsar timing code, and I have been trying to get erfa_python to work. I am currently on an OSX Mountain Lion (10.8.4) distro, and I am seeing some version mismatch between erfa_python and erfa. Note that I also tested it on a virtual machine with Debian, where everything works no problem. Extra info below.

Hope you can help! Thanks,

Rutger

The error I get:

(dev)[ vhaaster dhcp-195-218 ~/local/erfa ] python Python 2.7.5 (default, Oct 18 2013, 12:58:26) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.75)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import _erfa Traceback (most recent call last): File "", line 1, in ImportError: dlopen(/Users/vhaaster/env/dev/lib/python2.7/site-packages/_erfa.so, 2): Library not loaded: /Users/vhaaster/local/erfa/lib/liberfa.1.dylib Referenced from: /Users/vhaaster/env/dev/lib/python2.7/site-packages/_erfa.so Reason: Incompatible library version: _erfa.so requires version 2.0.0 or later, but liberfa.1.dylib provides version 0.0.0 exit()

I basically followed this instruction set to install erfa: export INSTDIR=/Users/vhaaster/local/ git clone https://github.com/liberfa/erfa-fetch.git cd erfa-fetch python sofaderiver.py cd erfa gcc -O2 -I. -c -fPIC .c gcc -shared -o liberfa.so *.o -lm cp erfa_.h $INSTDIR/include/ cp liberfa.so $INSTDIR/lib/liberfa.so.1.0.0 ln -sf $INSTDIR/lib/liberfa.so.1.0.0 $INSTDIR/lib/liberfa.dylib ln -sf $INSTDIR/lib/liberfa.so.1.0.0 $INSTDIR/lib/liberfa.1.dylib

[nirinA]

hi @vhaasteren thanks for the report. i cannot help much with some OSX platform as i haven't such machine . can you, please, try to download the SOFA tarball

http://www.iausofa.org/2013_1202_C/sofa_c-20131202.tar.gz

and then apply the script sofa_deriver.py to it, instead of liberfa from github. thanks

[nirinA]

@vhaasteren and a blind guess, may be, you need to name it liberfa.dylib instead of liberfa.so when creating the shared lib:

gcc -shared -o liberfa.dylib *.o -lm

[vhaasteren]

Hmm, I am about to give up for the day. It seems to be the version information that has in the erfa library. The one from the iausofa.org yields the same, incorrect, result. This is basically what seems to be wrong, the two libraries have incompatible version numbers:

(dev)[ vhaaster dhcp-195-218 ~/research/code/erfa_python (master) ] otool -L build/lib.macosx-10.8-x86_64-2.7/_erfa.so build/lib.macosx-10.8-x86_64-2.7/_erfa.so: /Users/vhaaster/local/erfa/lib/liberfa.1.dylib (compatibility version 2.0.0, current version 2.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

(dev)[ vhaaster dhcp-195-218 ~/research/code/erfa_python (master) ] otool -L /Users/vhaaster/local/erfa/lib/liberfa.1.dylib /Users/vhaaster/local/erfa/lib/liberfa.1.dylib: liberfa.so (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) (dev)[ vhaaster dhcp-195-218 ~/research/code/erfa_python (master) ]

This would mean that the library created from the erfa source has the wrong version number. But I cannot judge at the moment whether that is in erfa, or in the erfa_deriver...

[nirinA]

maybe, we need some additional info in setup.py to handle some Darwin/OSX platform. i'll take a look but need your help to test it.

[vhaasteren]

Sure, no problem, let me know what I can do. I'll ask around if anyone here can help.

[nirinA]

version mismatch makes me think that you may have multiple Python version on your system. also erfa_python and liberfa are, perhaps, not built with the same compiler/library.

[vhaasteren]

Thank you for your quick help, it is really appreciated.

I just checked your suggestion. I recompiled both erfa and erfa_python with another compiler. Initially both used Apple LLVM (clang-500.2.76). Now I switched to gcc 4.9.0 (20131110). Basically I removed everything (source files, libraries, and files from Python site-packages in my virtualenv), set export CC=gcc-4.9, and re-did the whole thing. Same results: version mismatch.

Makes me wonder where the version of the library is actually set.

[nirinA]

next try is then to have a build script for liberfa inside setup.py.

• use : "erfa-fetch / source_flattener.py" to create a single erfa.c file
• put it and erfa[m].h in the same directory as setup.py
• with a Makefile like this:

    all : 
        gcc -Wall -O -fPIC -shared -o liberfa.dylib erfa.c

• and a setup.py like this:

from distutils.core import setup, Extension
import subprocess
subprocess.Popen('make')
INC_DIR = ['.']
LIB_DIR = ['.']

moduleerfa = Extension('_erfa',
                    include_dirs = INC_DIR,
                    libraries = ['erfa', 'm'],
                    library_dirs = LIB_DIR,
                    sources = ['src/_erfamodule.c'])

setup (name = 'erfa_python',
       py_modules=['erfa'],
       ext_modules = [moduleerfa, ])

maybe additional code to select the appropriate compiler that was used to compile Python.

[vhaasteren]

Thanks again!

Ok, I tried what you suggested. First off, the erfa.h file that is produced by source_flattener needs to be reorganised. The stuff that is usually in erfam.h is placed at the bottom of the file, but it should be at the top. I had to cut & paste that bit to make it compile. Otherwise you get the error that eraASTROM is not defined.

Otherwise, it seems the problem is still there. I get the same result, whether I use clang or gcc 4.9. Output pasted below.

Is it possible that this can be solved simply by giving the compiler an option that sets the 'soname' in some way? I don't have any experience with that, but after a few google sessions that caught my eye.

Thanks,

Rutger

(dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] rm ~/env/dev/lib/python2.7/site-packages/_erfa.so (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] rm liberfa.dylib (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] ls ERFA.rst LICENSE MANIFEST.in Makefile README.md build doc erfa.c erfa.h erfa.h~ erfa.py erfa.pyc erfa_test.py examples setup.py setup.py.old setup.py~ src (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] rm -rf build/* (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] python setup.py install gcc -Wall -O -fPIC -shared -o liberfa.dylib erfa.c running install running build running build_py creating build/lib.macosx-10.8-x86_64-2.7 copying erfa.py -> build/lib.macosx-10.8-x86_64-2.7 running build_ext building '_erfa' extension creating build/temp.macosx-10.8-x86_64-2.7 creating build/temp.macosx-10.8-x86_64-2.7/src cc -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/_erfamodule.c -o build/temp.macosx-10.8-x86_64-2.7/src/_erfamodule.o cc -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.8-x86_64-2.7/src/_erfamodule.o -L. -lerfa -lm -o build/lib.macosx-10.8-x86_64-2.7/_erfa.so running install_lib copying build/lib.macosx-10.8-x86_64-2.7/_erfa.so -> /Users/vhaaster/env/dev/lib/python2.7/site-packages running install_egg_info Removing /Users/vhaaster/env/dev/lib/python2.7/site-packages/erfa_python-0.0.0-py2.7.egg-info Writing /Users/vhaaster/env/dev/lib/python2.7/site-packages/erfa_python-0.0.0-py2.7.egg-info (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] otool -L liberfa.dylib liberfa.dylib: liberfa.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] otool -L ~/env/dev/lib/python2.7/site-packages/_erfa.so /Users/vhaaster/env/dev/lib/python2.7/site-packages/_erfa.so: /Users/vhaaster/local/erfa/lib/liberfa.1.dylib (compatibility version 2.0.0, current version 2.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] cp liberfa.dylib ~/local/lib/liberfa.dylib (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] ln -s ~/local/lib/liberfa.dylib ~/local/erfa/lib/liberfa.1.dylib (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] python Python 2.7.5 (default, Oct 18 2013, 12:58:26) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.75)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import erfa Traceback (most recent call last): File "", line 1, in File "erfa.py", line 2, in from _erfa import * ImportError: dlopen(/usr/local/lib/python2.7/site-packages/_erfa.so, 2): Library not loaded: /Users/vhaaster/local/erfa/lib/liberfa.1.dylib Referenced from: /usr/local/lib/python2.7/site-packages/_erfa.so Reason: Incompatible library version: _erfa.so requires version 2.0.0 or later, but liberfa.1.dylib provides version 0.0.0 exit() (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ]

[nirinA]

what strikes me is that the Import Error states:

ImportError: dlopen(/usr/local/lib/python2.7/site-packages/erfa.so, 2):

but the setup.py installs things in:

running installlib copying build/lib.macosx-10.8-x86_64-2.7/erfa.so -> /Users/vhaaster/env/dev/lib/python2.7/site- packages

so at some point, there are two versions of _erfa.so in your system. can you try to rebuild erfa_python by explicitly name the interpreter, something like:

/usr/local/bin/python2.7 setup.py install

or

/Users/vhaaster/env/dev/bin/python setup.py install

then run the same interpreter to import erfa?

[nirinA]

in your fist report, the ImportError is:

ImportError: dlopen(/Users/vhaaster/env/dev/lib/python2.7/site-packages/erfa.so, 2):

which lets me a bit confused

[vhaasteren]

I managed to get it to work. It was indeed the soname of the library as I thought. When I create liberfa.dylib with the following command, it has the right version:

gcc -Wall -O -fPIC -shared -Wl,-compatibility_version,2.0.0,-current_version,2.0.0 -o liberfa.dylib erfa.c

Now: (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] gcc -Wall -O -fPIC -shared -Wl,-compatibility_version,2.0.0,-current_version,2.0.0 -o liberfa.dylib erfa.c (dev)[ vhaaster LMC-022025 ~/research/code/erfa_python (master) ] otool -L liberfa.dylib liberfa.dylib: liberfa.dylib (compatibility version 2.0.0, current version 2.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

[nirinA]

that's great! i'll add this issue in the next commit, for OSX users. thank you very much.

[vhaasteren]

No problem. And thank you too for the support!

[nirinA]

hi Rutger, can you, please, try if setting these versions compilation flags also work on the erfa_python side? that is, after building liberfa, build erfa_python with setup.py like the following:

import  sys
from distutils.core import setup, Extension
INC_DIR = ['$HOME/include',
           '/usr/include',
           '/usr/local/include']
LIB_DIR = ['$HOME/lib',
           '/usr/lib64',
           '/usr/lib',
           '/usr/local/lib',]

extra_compile_args = []
if sys.platform == 'darwin':
    extra_compile_args.append('-Wl,-compatibility_version,0.0.0,-current_version,0.0.0')
    LIB_DIR.append('/Users/vhaaster/local/erfa/lib')
    INC_DIR('/Users/vhaaster/local/erfa/include')

moduleerfa = Extension('_erfa',
                    include_dirs = INC_DIR,
                    libraries = ['erfa', 'm'],
                    library_dirs = LIB_DIR,
                    extra_compile_args = extra_compile_args,
                    sources = ['src/_erfamodule.c'])

setup (name = 'erfa_python',
       py_modules=['erfa'],
       ext_modules = [moduleerfa, ])

[nirinA]

oops,

INC_DIR('/Users/vhaaster/local/erfa/include')

needs to be:

INC_DIR.append('/Users/vhaaster/local/erfa/include')

[vhaasteren]

Sorry for the delay.

The setup.py script seems slightly faulty. The args should be linking args, as in 'extra_link_args', and the option '-Wl' is given to gcc to indicate it should be a list passed to the linker. So we need to create the list as follows:

extra_link_args = [] if sys.platform == 'darwin': extra_link_args.extend(['-compatibility_version', '0.0.0', \ '-current_version', '0.0.0']) LIB_DIR.append('/Users/vhaaster/local/erfa/lib') INC_DIR.append('/Users/vhaaster/local/erfa/include')

moduleerfa = Extension('_erfa', include_dirs = INC_DIR, libraries = ['erfa', 'm'], library_dirs = LIB_DIR, extra_link_args = extra_link_args, sources = ['src/_erfamodule.c'])

However, this does not compile it for me yet, and for the moment I cannot make more time to figure it out. This is the output:

[ vhaaster dhcp-195-161 ~/research/code/erfa_python (master) ] python setup.py install running install running build running build_py creating build/lib.macosx-10.8-x86_64-2.7 copying erfa.py -> build/lib.macosx-10.8-x86_64-2.7 running build_ext building '_erfa' extension creating build/temp.macosx-10.8-x86_64-2.7 creating build/temp.macosx-10.8-x86_64-2.7/src clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I$HOME/include -I/usr/include -I/usr/local/include -I/Users/vhaaster/local/erfa/include -I/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/_erfamodule.c -o build/temp.macosx-10.8-x86_64-2.7/src/_erfamodule.o clang -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.8-x86_64-2.7/src/_erfamodule.o -L$HOME/lib -L/usr/lib64 -L/usr/lib -L/usr/local/lib -L/Users/vhaaster/local/erfa/lib -lerfa -lm -o build/lib.macosx-10.8-x86_64-2.7/_erfa.so -compatibility_version 0.0.0 -current_version 0.0.0 clang: error: invalid argument '-compatibility_version 0.0.0' only allowed with '-dynamiclib' error: command 'clang' failed with exit status 1

Can do more later. Cheers,

Rutger

[nirinA]

Le Wed, 19 Feb 2014 04:18:51 +0300, Rutger notifications@github.com a
écrit:

Sorry for the delay.

no problem.

The setup.py script seems slightly faulty. The args should be linking
args, as in 'extra_link_args', and the option '-Wl' is given to gcc to
indicate it should be a list passed to the linker. So we need to create
the list as follows:

extra_link_args = [] if sys.platform == 'darwin': extra_link_args.extend(['-compatibility_version', '0.0.0', \ '-current_version', '0.0.0']) LIB_DIR.append('/Users/vhaaster/local/erfa/lib') INC_DIR.append('/Users/vhaaster/local/erfa/include')

moduleerfa = Extension('_erfa', include_dirs = INC_DIR, libraries = ['erfa', 'm'], library_dirs = LIB_DIR, extra_link_args = extra_link_args, sources = ['src/_erfamodule.c'])

However, this does not compile it for me yet, and for the moment I
cannot make more time to figure it out. This is the output:

[ vhaaster dhcp-195-161 ~/research/code/erfa_python (master) ] python
setup.py install running install running build running build_py creating build/lib.macosx-10.8-x86_64-2.7 copying erfa.py -> build/lib.macosx-10.8-x86_64-2.7 running build_ext building '_erfa' extension creating build/temp.macosx-10.8-x86_64-2.7 creating build/temp.macosx-10.8-x86_64-2.7/src clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include
-I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -I$HOME/include -I/usr/include -I/usr/local/include
-I/Users/vhaaster/local/erfa/include
-I/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/include/python2.7
-c src/_erfamodule.c -o
build/temp.macosx-10.8-x86_64-2.7/src/_erfamodule.o clang -bundle -undefined dynamic_lookup -L/usr/local/lib
-L/usr/local/opt/sqlite/lib
build/temp.macosx-10.8-x86_64-2.7/src/_erfamodule.o -L$HOME/lib
-L/usr/lib64 -L/usr/lib -L/usr/local/lib
-L/Users/vhaaster/local/erfa/lib -lerfa -lm -o
build/lib.macosx-10.8-x86_64-2.7/_erfa.so -compatibility_version 0.0.0
-current_version 0.0.0 clang: error: invalid argument '-compatibility_version 0.0.0' only
allowed with '-dynamiclib' error: command 'clang' failed with exit status 1

so, it's better to left the setup script as is for now, and correct this compatibility version in the erfa library.

Can do more later. Cheers,

don't worry, this is not an urgent issue and i also have no much time for now. thank you very much for your time.

best regards,

Rutger

---

Reply to this email directly or view it on GitHub: https://github.com/nirinA/erfa_python/issues/4#issuecomment-35456092

nirinA

[nirinA]

hi @vhaasteren , i pushed a new commit and add a note for building on OSX, on doc/installation.rst. please fell free to correct any mistake.

[nirinA]

@vhaasteren , @scottransom i've included erfa source so that installing the liberfa package is no longer required! can you, please, test if the new setup work, especially for OSX platform ? with this new version, compiling liberfa can be avoided when using PINT.

thanks

[vhaasteren]

Hi nirinA,

Thank you for that work. However: are you suggesting that I no longer would need to have an erfa.so or similar installed? Because that does not work for me (see below). It seems the installer does not create the _erfa.so that is being referred to. Or am I still required to install that separately (in which case I misunderstood).

Thanks!

Output:

(dev)[ vhaaster dhcp-195-184 ~/research/code/erfa_python (master) ] python setup.py build running build running build_py running build_ext (dev)[ vhaaster dhcp-195-184 ~/research/code/erfa_python (master) ] python setup.py install running install running build running build_py running build_ext running install_lib running install_egg_info Removing /Users/vhaaster/env/dev/lib/python2.7/site-packages/erfa_python-2014.06.02-py2.7.egg-info Writing /Users/vhaaster/env/dev/lib/python2.7/site-packages/erfa_python-2014.06.02-py2.7.egg-info (dev)[ vhaaster dhcp-195-184 ~/research/code/erfa_python (master) ] python Python 2.7.5 (default, Oct 18 2013, 12:58:26) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.75)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import erfa Traceback (most recent call last): File "", line 1, in File "erfa.py", line 2, in from _erfa import * ImportError: dlopen(/usr/local/lib/python2.7/site-packages/_erfa.so, 2): Library not loaded: /Users/vhaaster/local/erfa/lib/liberfa.1.dylib Referenced from: /usr/local/lib/python2.7/site-packages/_erfa.so Reason: image not found exit()

[nirinA]

hi Rutger, thank you for testing the package. sorry to hear that you are still in trouble with the new setup. with the erfa.c provided, you should no longer need some liberfa installed. on my linux box, for example, python setup.py install: create _erfamodule.o and erfa.o objects, then _erfa.so and install it with erfa.py in site-packages. and the module works with no liberfa pre-installed.

maybe, on OSX, the erfa.c source code needs to be compiled as a library?

can you, please, try to build the module manually, instead of using the setup.py script? something like:

gcc -pthread -O2 -DNDEBUG -fPIC -Isrc -I/usr/include/python2.7 -c src/erfa.c -o src/erfa.o
gcc -pthread -O2 -DNDEBUG -Wall -fPIC -Isrc -I/usr/include/python2.7 -c src/_erfamodule.c -o src/_erfamodule.o
gcc -pthread -shared src/erfa.o src/_erfamodule.o -L/usr/local/lib -lm -lpython2.7 -o _erfa.so

and add to sys.path the location of _erfa.so before invoking import _erfa

i'm really sorry, i cannot help much as i never use some osx machine.

[nirinA]

hi Rutger, please disregard my last request as you already did that. also, for serious usage, please consider using the astropy package as erfa is now included in it, unless you have to use erfa somewhere without the need of installing anything else than Python. best regards,

[vhaasteren]

Thank you for getting back at me nirinA,

I have indeed discovered that erfa is now included in Astropy (through PINT), so that indeed solves all of my problems. Thank you very much for all your efforts.

Best regards,

Rutger