Problem finding libraries on OS X El Capitan

[mjuric]

@cwwalter reports:

Hi Guys,

OK install seems to have gone fine. I can run GalSim (which is good!)

When I try to run

https://github.com/cwwalter/brighter-fatter/blob/master/processSpots.py

after setting up pipe_tasks (which sets up afw)

It’s failing. ——-

ImportError Traceback (most recent call last) /Users/walter/LSST/brighter-fatter/processSpots.py in () 9 import itertools 10 ---> 11 import lsst.afw.math as afwMath 12 import lsst.afw.table as afwTable 13 import lsst.afw.image as afwImg

/Users/walter/LSST/Anaconda/opt/lsst/afw/python/lsst/afw/math/init.py in () 23 """Application Framework math code including Function, minimize, Kernel and convolve 24 """ ---> 25 from mathLib import 26 from warper import 27 from background import *

/Users/walter/LSST/Anaconda/opt/lsst/afw/python/lsst/afw/math/mathLib.py in () 32 fp.close() 33 return _mod ---> 34 _mathLib = swig_import_helper() 35 del swig_import_helper 36 else:

/Users/walter/LSST/Anaconda/opt/lsst/afw/python/lsst/afw/math/mathLib.py in swig_import_helper() 28 if fp is not None: 29 try: ---> 30 _mod = imp.load_module('_mathLib', fp, pathname, description) 31 finally: 32 fp.close()

/Users/walter/LSST/Anaconda/opt/lsst/base/python/lsstimport.pyc in imp_load_module(name, args) 100 sys.setdlopenflags(DLFLAGS) 101 try: --> 102 module = orig_imp_load_module(name, args) 103 finally: 104 # Set original flags

ImportError: dlopen(/Users/walter/LSST/Anaconda/opt/lsst/afw/python/lsst/afw/math/_mathLib.so, 10): Library not loaded: @rpath/libboost_system.dylib Referenced from: /Users/walter/LSST/Anaconda/opt/lsst/boost/lib/libboost_filesystem.dylib Reason: image not found

——

For what it is worth. That file is there:

wasabi:LSST % ls Anaconda/opt/lsst/afw/python/lsst/afw/math/_mathLib.so Anaconda/opt/lsst/afw/python/lsst/afw/math/_mathLib.so*

I am sourcing:

/Users/walter/LSST/Anaconda/bin/eups-setups.csh

has something changed so that I need to do something else?

-Chris

[mjuric]

@cwwalter, @danielsf Just FYI, I tried reproducing this on Yosemite, and there it works fine. This is an El Capitan problem. Will try to debug it -- I don't have an access to a machine with El Capitan, so I'll see if I can get it installed in a VM instead.

[mjuric]

@cwwalter -- how did you run your processSpots.py file? Just with python processSpots.py?

[cwwalter]

I ran it with iPython (from the conda distribution).

[cwwalter]

so 'ipython processSpots.py'

I am using the latest version of anaconda (2.4) if that could make a difference.

[cwwalter]

Um this is weird.

I just tried running with python instead of ipython and it works.

How could that be?!

[cwwalter]

So from the ipython interpreter prompt just typing

import lsst.afw.math

gives the error.

ImportError: dlopen(/Users/walter/LSST/Anaconda/opt/lsst/afw/python/lsst/afw/math/_mathLib.so, 10): Library not loaded: @rpath/libboost_system.dylib Referenced from: /Users/walter/LSST/Anaconda/opt/lsst/boost/lib/libboost_filesystem.dylib Reason: image not found

[mjuric]

Interesting. I suspect ipython starts its kernels by invoking them via /bin/sh, which wipes out the DYLD_LIBRARY_PATH environment variable. Try this for an experiment:

python -c 'import os; print os.environ["DYLD_LIBRARY_PATH"];'

and then

ipython -c 'import os; print os.environ["DYLD_LIBRARY_PATH"];'

Any difference?

[cwwalter]

Yes, very different. It looks like you are right.

'python' gives a giant path including all the LSST modules.

'ipython' gives:

wasabi:walter % ipython -c 'import os; print os.environ["DYLD_LIBRARY_PATH"];'

KeyError Traceback (most recent call last)

in () ----> 1 import os; print os.environ["DYLD_LIBRARY_PATH"]; /Users/walter/LSST/Anaconda/lib/python2.7/UserDict.pyc in **getitem**(self, key) 38 if hasattr(self.**class**, "**missing**"): 39 return self.**class**.**missing**(self, key) ---> 40 raise KeyError(key) 41 def **setitem**(self, key, item): self.data[key] = item 42 def **delitem**(self, key): del self.data[key] KeyError: 'DYLD_LIBRARY_PATH'

[mjuric]

I hate being right on this one. It'll make it a (potentially very) tricky fix.

Need to retreat to my hole and think...

[cwwalter]

I think this is going to be a quite general problem right? People trying to run MAF in ipython notebooks are going to have this problem too.

[rhiannonlynne]

MAF won't have the same problem with most stuff, because we're all python. The camera geometry related stuff might be problematic but I'm not sure that's not pure python too?? (we might end up running in C++ stuff at WCS, but I'm not sure if we're using that or not).

On Wed, Dec 30, 2015 at 11:24 AM Chris Walter notifications@github.com wrote:

I think this is going to be a quite general problem right? People trying to run MAF in ipython notebooks are going to have this problem too.

— Reply to this email directly or view it on GitHub https://github.com/mjuric/conda-lsst/issues/33#issuecomment-168049218.

[rhiannonlynne]

Hmm. Although, on second thought, we might have some issues with healpy / healpix? I'll have to update to El Capitan and see I guess.

On Wed, Dec 30, 2015 at 11:59 AM Lynne Jones ljones.uw@gmail.com wrote:

MAF won't have the same problem with most stuff, because we're all python. The camera geometry related stuff might be problematic but I'm not sure that's not pure python too?? (we might end up running in C++ stuff at WCS, but I'm not sure if we're using that or not).

On Wed, Dec 30, 2015 at 11:24 AM Chris Walter notifications@github.com wrote:

I think this is going to be a quite general problem right? People trying to run MAF in ipython notebooks are going to have this problem too.

— Reply to this email directly or view it on GitHub https://github.com/mjuric/conda-lsst/issues/33#issuecomment-168049218.

[mjuric]

@cwwalter @rhiannonlynne Yes, this will be a major issue for everyone, as it's hard to avoid C/C++ modules down below (and you wouldn't want your users to have to think about it). I think I know how to fix it in conda-land (need to think more). There's also a possibility that this is a simpler bug than I thought (a real conda packaging bug, instead of an architectural issue -- I'm looking into it).

I'll ping @timj for the regular builds. Tim, I think the summary is that in El Capitan we can't rely on DYLD_LIBRARY_PATH at all (because ipython may not heed the various warnings from dmtn-001 and may wipe DYLD_LIBRARY_PATH out). I fear that everything may need to be properly @rpath-ed (which will effectively disable the use of EUPS' mix-and-match functionality on El Capitan).

[timj]

Yes, there was a whole discussion on HipChat before Christmas about iPython and El Capitan and DYLD_LIBRARY_PATH. That triggered this response on Stack Overflow:

See http://stackoverflow.com/questions/32949071/ipython-cant-load-a-module-which-python-does/34168933#34168933

[mjuric]

@timj Thanks. That's what I feared.

It'd be good to have an 'El Capitan Issues' page on community (or JIRA?), recording known issues (HipChat discussions are considered ephemeral).

[ebellm]

Just to confirm: I upgraded to El Capitan this week and conda-installed the maf stack. Indeed, it fails simple imports (e.g., import lsst.sims.maf.slicers as slicers) due to the DYLD_LIBRARY_PATH issue.

As a workaround, I disabled the El Capitan System Integrity Protection, and everything appears to work fine.

[mjuric]

Hi @ebellm, yes, El Capitan's SIP has essentially completely wrecked the method we've used to load dynamic libraries -- it can't be robustly done any more (e.g., to ensure IPython compatibility), and I suspect a major rethink may be needed there.

For those on El Cap, disabling SIP is probably the best workaround.

[cwwalter]

The latest versions of the stack have a fix for this (implemented by @timj). He can point you to the LSST document that describes it. I'm not sure if it is the permanent solution (in regards to Mario's comments he probably knows better than me!). But Mario's dev version of the conda stack install includes these fixes and it works on El Capitan. You don't need to turn off SIP.

[cwwalter]

Forgot to say I am using it and can confirm it works.

[mjuric]

@cwwalter Right, with @timj's (heroic) work, running from the command line will work, but using it from IPython is still a problem, isn't it? The issue you reported here -- I'd be surprised if that went away?

[cwwalter]

Oh drat... I forgot all about my own report! Lame!

yes... it still doesn't work from the ipython interpreter. It's OK from the anaconda python itself. It shows up for things that use c++ parts of the code base.

[cwwalter]

For the record here is how ipython is started.

#!/bin/bash /Users/walter/LSST/Anaconda/bin/python.app
if __name__ == '__main__':
    import sys
    from IPython import start_ipython

    sys.exit(start_ipython())

[cwwalter]

So just for fun: If you start up python and then put in:

import sys from IPython import start_ipython sys.exit(start_ipython())

it starts up the interpreter and you can import afw etc correctly.

[mjuric]

Hah, that's interesting! It's also something the Jupyter folks may be willing to fix for us.

[timj]

As I say in my stackoverflow response mentioned above, the fix is simple but I'm not sure why iPython uses bash so I'm not sure what the justification is for having it the way it is.

[mjuric]

Right, plus it's unlikely to be robust -- the first time someone fires off system() to launch something that uses the stack, game over. Methinks envvars are out, and linkfarms are in (or just conda environments, in this particular case).

How far are we from being able to run the stack w/o EUPS environmental variables (the *_DIR ones)? I think that was the primary blocker for the pure link farm (and EUPS-less running) solution.

[timj]

The stack does not rely on eups at all (assuming relevant environment variables are set for PYTHONPATH and the like). The stack tests have all been rewritten to stop using eups and to start using the lsst.utils.getPackageDir and that relies on the *_DIR environment variables existing. In theory any new scheme should be able to just reimplement that function (and anything that is using *_DIR explicitly should just be fixed).

[mjuric]

Awesome!

Hmm, I notice Monday is a holiday, meaning I may have some time to work on this... Somebody ping me on Tuesday :).

[cwwalter]

In a conda link like environment, is it possible to mix versions of different packages? Or only whole sets of environments? That seems to be the thing that is most important to people in EUPS.

Of course we know it is possible at some level since I did it by mistake with GalSim yesterday :)

[mjuric]

@cwwalter @rhiannonlynne @timj: An interesting discovery; while the conda-supplied ipython script is invoked with #!/bin/bash, the jupyter script isn't.

Could anyone try if opening your notebook with jupyter makes import lsst.afw.math pass?

[timj]

https://community.lsst.org/t/library-loading-error-in-an-ipython-notebook/623

[mjuric]

Ah, awesome! I remember Lynne trying it out a few days ago, but somehow I thought she also had her ipython with the shebang fixed?

[mjuric]

Btw., the upstream issue is at https://github.com/conda/conda-build/issues/628.

[rhiannonlynne]

If you have a very new (and clean) anaconda install, you actually don't get an option to do ipython notebook -you have to do jupyter notebook. This is the direction jupyter is going, I think.

On Sun, Apr 3, 2016, 9:34 PM Mario Juric notifications@github.com wrote:

Btw., the upstream issue is at conda/conda-build#628 https://github.com/conda/conda-build/issues/628.

— You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub https://github.com/mjuric/conda-lsst/issues/33#issuecomment-205133363

[cwwalter]

Sorry! I thought I had pointed this out somewhere else.

Also, you can run 'juptyer console' to bring up an interactive ipython shell that works to import afw etc.

[mjuric]

Folks: the Continuum guys were lightning fast and new ipython packages with the fixed shebang are out.

Could someone on ElCap do conda update ipython and check out if all is well now?

[rhiannonlynne]

Looks like it should be.

#!/Users/lynnej/anaconda/bin/python
if __name__ == '__main__':
    import sys
    import IPython

    sys.exit(IPython.start_ipython())

[mjuric]

Just to verify before declaring victory -- does import LSST.afw.math work from within ipython shell?

[timj]

Yes it does.

[mjuric]

Excellent, thanks! I've added a note to the associated c.l.o post and will be closing this issue. Reopen if need be.