PySide in relocatable virtualenv: Library not loaded: @rpath/libpyside-python2.7.1.2.dylib

[fredrikaverpil]

I'm pip installing PySide into a virtualenv:

sudo virtualenv --always-copy $VENV
source $VENV/bin/activate
sudo $VENV/bin/pip install pyside
sudo $VENV/bin/python $VENV/bin/pyside_postinstall.py -install
deactivate

The virtualenv works great and I can run my Python/PySide script without problems:

$VENV/bin/python $SCRIPT

Now, I make my virtualenv relocatable:

virtualenv --relocatable $VENV

I move it:

mv $VENV $VENV_RELOCATED

...and again run my script:

source $VENV_RELOCATED/bin/activate
$VENV_RELOCATED/bin/python $SCRIPT

This time around, I'm getting an error:

Traceback (most recent call last):
  File "script.py", line 1, in <module>
    from PySide import QtCore, QtGui, QtUiTools
ImportError: dlopen(/absolute/path/to/relocated_venv/lib/python2.7/site-packages/PySide/QtCore.so, 2): Library not loaded: @rpath/libpyside-python2.7.1.2.dylib
  Referenced from: /absolute/path/to/relocated_venv/lib/python2.7/site-packages/PySide/QtCore.so
  Reason: image not found

These are the contents of the script:

from PySide import QtCore, QtGui, QtUiTools

Please note: the absolute path printed twice in the error message is the $VENV_RELOCATED

This is all on OS X 10.11.0 with Python 2.7.10 and PySide 1.2.2.

Any ideas on workaround so that the virtualenv could be moved around?

[fredrikaverpil]

I made PySide work inside a relocatable virtualenv on OS X 10.11 (El Capitan).

I modified the localize_libpaths() function in the pyside_postinstall.py script. If a virtualenv is detected, it will not use @rpath. Instead it will use @executable_path and search relatively from the python binary in the virtualenv. Since the PySide module is always installed in the same place within a virtualenv, I thought this should work in all scenarios. I believe this function is only run on darwin systems... But I will now also look into running PySide in a relocatable virtualenv on Windows 10 and as well as on Ubuntu 14.04. But it works great so far on OS X 10.11 (El Capitan).

I don't know if you think this is a proper fix/workaround for OS X?

def localize_libpaths(libpath, local_libs, enc_path=None):
    """ Set rpaths and install names to load local dynamic libs at run time

    Use ``install_name_tool`` to set relative install names in `libpath` (as
    named in `local_libs` to be relative to `enc_path`.  The default for
    `enc_path` is the directory containing `libpath`.

    Parameters
    ----------
    libpath : str
        path to library for which to set install names and rpaths
    local_libs : sequence of str
        library (install) names that should be considered relative paths
    enc_path : str, optional
        path that does or will contain the `libpath` library, and to which the
        `local_libs` are relative.  Defaults to current directory containing
        `libpath`.
    """
    if enc_path is None:
        enc_path = abspath(dirname(libpath))
    install_names = osx_get_install_names(libpath)
    need_rpath = False
    for install_name in install_names:
        if install_name[0] in '/@':
            continue
        if hasattr(sys, 'real_prefix'):
            # virtualenv detected - use @executable_path
            back_tick('install_name_tool -change %s @executable_path/../lib/python2.7/site-packages/PySide/%s %s' %
               (install_name, install_name, libpath))
        else:
            # not a virtualenv - use @rpath
            back_tick('install_name_tool -change %s @rpath/%s %s' %
               (install_name, install_name, libpath))
            need_rpath = True
    if need_rpath and enc_path not in osx_get_rpaths(libpath):
        back_tick('install_name_tool -add_rpath %s %s' %
           (enc_path, libpath))

[ctismer]

Interesting, but did you use the latest version? I mean, there is no version, the top has not yet been tagged, but that version does not have a pyside_postinstall.py any longer. If you check out the top version and build pyside with --ignore-git, after applying your patch, then it makes sense to discuss it further.

Actually, I'm interested to use it on PySide2, that is the upcoming Qt5 version.

[ctismer]

Ah, you used version 1.2.2, because that for-loop from above has a bug, that I fixed in Pyide and PySide2.

[fredrikaverpil]

If you check out the top version and build pyside with --ignore-git, after applying your patch, then it makes sense to discuss it further.

Ok, I'll give it a shot. Do you mean like this?

git clone --recursive https://github.com/PySide/pyside-setup.git pyside-setup
python setup.py bdist_wheel --ignore-git

I'm actually getting this when cloning pyside-setup like that:

fatal: reference is not a tree: aa39374b419c535dd56145ffebcef97c8c20eb39
Unable to checkout '0c64d1b2c6e5e0951675ad9b22294db4a10741c7' in submodule path 'sources/pyside'
Unable to checkout 'aa39374b419c535dd56145ffebcef97c8c20eb39' in submodule path 'sources/shiboken'

Any idea on how I should proceed?

[ctismer]

Oops, I just saw that I made a bad change, from the old gitorious submodules to github, but the modules were old ones from 2012. Will fix that, sorry

[fredrikaverpil]

Great, thanks. Let me know when it's ready. I'm calling it a day now :)

[fredrikaverpil]

By the way, I notice that another patch is needed to make PySide work properly in a relocatable virtualenv on Linux. I'm not sure if patchelf can be used to set something similar to dyld's @executable_path on OSX, rather than setting rpath via patchelf --set-rpath ...?

On Windows it all works as-is with no need for any patching.

[ctismer]

Ok, I think it is ready. Restored things with the help of PySide2. Then I had to search a while until I found out how to get rid of the Qt5 branches: I needed to also removed the tagging, then things finally vanished ;-)

[ctismer]

No idea about patchelf. I have to say that I always cared about OS X. Just since May, when I was forced to work with Windows and Linux on Qt5, but relocation was not my top priority.

If you find a nice solution, I'm happy to try a PR.

[fredrikaverpil]

I just compiled on both OS X and Linux and there's no need for any patching with this 1.3.0dev version of PySide. PySide works just fine even after having moved around the virtualenv. Awesome! :)

Is this 1.3.0dev version considered stable, or are you still fixing things and will release a true 1.3.0 later on?

About patchelf, you're supposed to be able to use $ORIGIN in the rpath which will make libraries reachable relative to the executable: http://stackoverflow.com/questions/3015411/shipping-gnu-linux-firefox-plugin-with-shared-libraries-for-installation-with-n – haven't tried this myself, though.

[ctismer]

Fine to hear that, good that nothing was permanently damaged. Actually, I did not want to do the final tagging, but was waiting for others who build the wheels, etc. And I guess this will just be 1.2.3, as the last Qt4 version. Then we will switch over to Qt5 (and this is a project that I first feared it would never start, and now I'm afraid it would never end), but that is still in alpha.

But maybe you motivated me to push it a bit ;-)

[fredrikaverpil]

Hehe, yeah. PySide is actually the standard in my line of work (visual effects/3D animation for films/commercials/archviz) so lots of people are dependent on it. – We have actually been discussing PySide vs PyQt for the Qt5 bindnings... but now it seems it's happening for PySide which is great news! :) However, I hope that pyside2 will work with Python 2.x ...?

By the way, I was going to build 1.3.0dev from git on Windows 10, and I followed these instructions. I ran this command in a git cmd prompt:

python setup.py bdist_wheel --ignore-git --qmake=c:\Qt\4.8.6\bin\qmake.exe --openssl=c:\OpenSSL-Win64\bin --cmake="C:\Program Files (x86)\CMake\bin\cmake.exe"

However, I got this back:

Removing C:\Users\fredrik\Desktop\virtualenvs\pyside-setup\pyside_package
running bdist_wheel
running build
Python architecture is 64bit
nmake not found. Trying to initialize the MSVC env...
Searching MSVC compiler version 9.0
error: Failed to find the MSVC compiler version 9.0 on your system.

The instructions doesn't say anything about Visual Studio 2008. Is it required or could I use something "lighter" than VS2008?

[lck]

Only Windows sdk is required

R. Dňa 3.10.2015 9:36 používateľ "Fredrik Averpil" notifications@github.com napísal:

Hehe, yeah. PySide is actually the standard in my line of work (visual effects for films/commercials/archviz) so lots of people are dependent on it even if it's in alpha :)

By the way, I was going to build from git on Windows 10, and I followed the instructions http://pyside.readthedocs.org/en/latest/building/windows.html. However, I got this:

C:\Users\fredrik\Desktop\virtualenvs\pyside-setup>python.exe setup.py bdist_wheel --ignore-git --qmake=c:\Qt\4.8.6\bin\qmake.exe --openssl=c:\OpenSSL-Win64\bin --cmake="C:\Program Files (x86)\CMake\bin\cmake.exe" Removing C:\Users\fredrik\Desktop\virtualenvs\pyside-setup\pyside_package running bdist_wheel running build Python architecture is 64bit nmake not found. Trying to initialize the MSVC env... Searching MSVC compiler version 9.0 error: Failed to find the MSVC compiler version 9.0 on your system.

The instructions doesn't say anything about Visual Studio 2008. Is it required?

— Reply to this email directly or view it on GitHub https://github.com/PySide/PySide/issues/129#issuecomment-145211943.

[fredrikaverpil]

Only Windows sdk is required

I did install Windows SDK 7 (on Windows 10), but it doesn't work using the command I posted, as you can see from the output: error: Failed to find the MSVC compiler version 9.0 on your system. Any idea what I am doing wrong?

Also, I'm on Windows 10 with 64-bit Python 2.7.9. Maybe I need the Windows SDK 10 and/or some specials to compile using 64-bit Python?

[lck]

You can try to run the build from sdk command prompt Dňa 3.10.2015 10:40 používateľ "Fredrik Averpil" notifications@github.com napísal:

Only Windows sdk is required

I did install Windows SDK 7, but how do I define nmake when building?

Also, I'm on Windows 10. Maybe I need the Windows SDK 10?

— Reply to this email directly or view it on GitHub https://github.com/PySide/PySide/issues/129#issuecomment-145216318.

[ctismer]

@fredrikaverpil btw., why are you using PySide and not PyQt? They have changed their licensing. Would be interesting to know (I just like PySide for a reason).

[fredrikaverpil]

@lck ah, that's probably it. Thanks. Will try this when I'm at the machine.

@ctismer actually, I don't have a preference. But since it's bundled with Autodesk's Maya and The Foundry's Nuke it works right out of the box in those applications.

[ctismer]

@fredrikaverpil interesting. That somehow closes the circle, because I also tried to learn why Autodesk insisted in PySide2 for Qt5. Some things simply persist, because of kinda virtual dependency. I like that.

[fredrikaverpil]

@ctismer I wonder if maybe Autodesk originally went with PySide because of licensing reasons. And now that they have a rather large user base with PySide scripts running in their software they may want to keep on going with PySide2... Are you working with them on PySide2?

I'm guessing that The Foundry also would like to have PySide2 bundled with Nuke since it comes with PySide today. Have you been in touch with The Foundry?

By the way, will PySide2 be backwards-compatible with PySide? Should these transitions also apply for when going from PySide to PySide2?

[fredrikaverpil]

I can now also confirm that building from github source worked just fine in Windows too (thanks @lck for the reminder on using the proper CMD shell). I compiled for 64-bit and thus I needed Qt4 64-bit. Rather than building it myself from Qt4 v4.8.7, I used these Qt v4.8.4 64-bit precompiled binaries (qt-win64-opensource-4.8.4-noqt3-vs2008). Would you say this is wrong and that I should compile against Qt4 v4.8.7? – I'm not familiar enough with what aspects of Qt are used during PySide compile to judge whether this is fine or not.

This was my build command:

python setup.py bdist_wheel --ignore-git --qmake=c:\Qt\4.8.4_x64\bin\qmake.exe --openssl=c:\OpenSSL-Win64\bin --cmake="C:\Program Files (x86)\CMake\bin\cmake.exe"

Just like with PySide 1.2.2 there are no issues with relocating the virtualenv with PySide 1.3.0dev in it.

To sum it up: I now have three PySide 1.3.0dev .whl files, one for each OS: Linux, OS X, Windows. They all install fine (pip) into a virtualenv and PySide still works properly after having moved the virtualenv around.

So in a way, I guess we could close this issue, as the next release of PySide (probably tagged v1.2.3) will address this. The issue remains with PySide 1.2.2.

[ctismer]

Would you say this is wrong and that I should compile against Qt4 v4.8.7?

No, I believe you are just fine with the pre-built build.

Yes, the issue will be closed with 1.2.3, if the others agree.

I guess the reluctance with tagging is that we than would have to build many binaries at once in no time, and that is always quite an undertaking. Should not be a problem with a build-bot. I think we will get one on PySide2.

[ctismer]

@fredrikaverpil I would like to get back to you with something in private emails.

[fredrikaverpil]

Yes, the issue will be closed with 1.2.3, if the others agree.

Sounds great!

@ctismer sure, my email is fredrik.averpil@gmail.com

[lck]

Yes, 1.2.3 should be released soon.

[ctismer]

I'm guessing that The Foundry also would like to have PySide2 bundled with Nuke since it comes with PySide today. Have you been in touch with The Foundry?

No, I have no idea what this is.

By the way, will PySide2 be backwards-compatible with PySide? Should these transitions also apply for when going from PySide to PySide2?

No, PySide2 is not backward compatible, and all the transitions apply. The transitions show only the tip of the iceberg - under the hood, the changes were even heavier, because PySide2 touches many internal things.

[duozmo]

Had this same issue with PySide 1.2.4. Installed via pip into an Anaconda environment, running on OS X 10.10.5 and Python 3.3. This came up while trying to use qt as a backend to matplotlib, though the line that triggers the error is basically the same as OP.

$ ipython
In [1]: %pylab qt

...
/absolute/path/to/anaconda_environment/lib/python3.3/site-packages/matplotlib/backends/qt_compat.py in <module>()
--> 126     from PySide import QtCore, QtGui, __version__, __version_info__
...

ImportError: dlopen(/absolute/path/to/anaconda_environment/lib/python3.3/site-packages/PySide/QtCore.so, 2): Library not loaded: @rpath/libpyside.cpython-33m.1.2.dylib
  Referenced from: /absolute/path/to/anaconda_environment/lib/python3.3/site-packages/PySide/QtCore.so
  Reason: image not found

If I fix the dynamic library path:

DYLD_LIBRARY_PATH=/absolute/path/to/anaconda_environment/lib/python3.3/site-packages/PySide/ ipython

Then the %pylab qt magic loads just fine.

[techtonik]

@duozmo I'd recommend opening a new ticket to not lose the info.

[duozmo]

Can't reopen?

[techtonik]

@duozmo I can't.

[ctismer]

ok, opened it again

[ctismer]

@duozmo let's reduce the number of variables: Can you create a simple virtualenv for a regular python and check whether pip then also complains? My idea is that it might be an anaconda problem.

[fredrikaverpil]

I also have this issue with PySide 1.2.4 when pip installing it into a virtualenv on OS X 10.11.

Macdev:virtualenvs fredrik$ pwd
/Users/fredrik/virtualenvs
Macdev:virtualenvs fredrik$ virtualenv --always-copy myenv
New python executable in myenv/bin/python2.7
Also creating executable in myenv/bin/python
Installing setuptools, pip, wheel...done.
Macdev:virtualenvs fredrik$ source myenv/bin/activate
(myenv)Macdev:virtualenvs fredrik$ pip install pyside
Collecting pyside
Installing collected packages: pyside
Successfully installed pyside-1.2.4
(myenv)Macdev:virtualenvs fredrik$ python
Python 2.7.10 (default, Nov  5 2015, 21:40:43) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PySide import QtCore, QtGui, QtUiTools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/fredrik/virtualenvs/myenv/lib/python2.7/site-packages/PySide/QtCore.so, 2): Library not loaded: @rpath/libpyside-python2.7.1.2.dylib
  Referenced from: /Users/fredrik/virtualenvs/myenv/lib/python2.7/site-packages/PySide/QtCore.so
  Reason: image not found
>>> 

[duozmo]

@ctismer Getting nearly an identical error in a simple virtualenv.

>>> from PySide import QtCore
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/path/to/pysideenv/lib/python3.4/site-packages/PySide/QtCore.so, 2): Library not loaded: @rpath/libpyside.cpython-34m.1.2.dylib
  Referenced from: /path/to/pysideenv/lib/python3.4/site-packages/PySide/QtCore.so
  Reason: image not found

That library is available, just apparently not where QtCore is looking:

pysideenv $ find . -iname libpyside*
./lib/python3.4/site-packages/PySide/libpyside.cpython-34m.1.2.4.dylib
./lib/python3.4/site-packages/PySide/libpyside.cpython-34m.1.2.dylib
./lib/python3.4/site-packages/PySide/libpyside.cpython-34m.dylib

Certainly seems like something to do with @rpath. I'm sorry I don't have more time to investigate. I'm not really a PySide user, I just wanted to stop by to report the error.

[techtonik]

I suspect that Linux doesn't allow to load libraries from arbitrary paths.

[lck]

Is this not macos spesific problem ? i have tested pyside in relacatable virtualenv on ubuntu mate without problems. Dňa 8.11.2015 9:46 používateľ "anatoly techtonik" notifications@github.com napísal:

I suspect that Linux doesn't allow to load libraries from arbitrary paths.

— Reply to this email directly or view it on GitHub https://github.com/PySide/PySide/issues/129#issuecomment-154789749.

[techtonik]

@lck I missed the OS X specifics, this issue need a flag that the bug is for OS X.

[fredrikaverpil]

Does anyone know of a workaround for this issue? There must be some way to make this work after having pip installed PySide?

[fredrikaverpil]

As long as Qt4 is installed (/usr/local/opt/qt/ in my case), you can do this as workaround:

$ export DYLD_LIBRARY_PATH=/Users/you/virtualenv/lib/python2.7/site-packages/PySide

$ /Users/you/virtualenv/python

dyld: warning, unknown environment variable: DYLD_LIBARY_PATH
Python 2.7.10 (default, Jul 14 2015, 19:46:27) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PySide import QtCore
>>> 

However, not very elegant.

[zzeleznick]

@fredrikaverpil This solved a similar problem for my root environment. I was able to just add the equivalent line to my .bash_profile and the "Reason: image not found" issue was resolved.

[shellyan]

@duozmo your method fixed my problem. my environment is OS X 10.11.1 (15B42).

[AwwCookies]

@fredrikaverpil thanks, that solved my issue.

[fredrikaverpil]

@AwwCookies @shellyan @zzeleznick I would like to stress that I don't think this workaround is a proper one. You shouldn't need to set DYLD_LIBRARY_PATH to make PySide work in this scenario.

[sagistrauss]

Hi guys, I'm still looking for a solution that doesn't include a workaround. Any ideas? still not working even with PySide 1.2.4. Same error:

from PySide import QtCore Traceback (most recent call last): File "", line 1, in ImportError: dlopen(/Library/Python/2.7/site-packages/PySide/QtCore.so, 2): Library not loaded: libpyside-python2.7.1.1.dylib Referenced from: /Library/Python/2.7/site-packages/PySide/QtCore.so Reason: unsafe use of relative rpath libpyside-python2.7.1.1.dylib in /Library/Python/2.7/site-packages/PySide/QtCore.so with restricted binary

[fredrikaverpil]

Hi guys, I'm still looking for a solution

+1

[mangopipeline]

I'm having the same issue, i've applied the patch above and was able to get it working on my dev machine, but obviously it's not possible to run it of the network unless other machines have qt installed... is there anyway to move the content of /usr/local/opt/qt/ to the virtualenv so that other machines don't have to do a brew install qt? cheers, Los

[sagistrauss]

So I ended up using Selenium with PhantomJS which is much better and doesn't have issues with libraries.. I recommend you do the same!

-S. ᐧ

On Thu, Mar 10, 2016 at 9:08 AM, Mangosoft LLC notifications@github.com wrote:

I'm having the same issue, i've applied the patch above and was able to get it working on my dev machine, but obviously it's not possible to run it of the network unless other machines have qt installed... is there anyway to move the content of /usr/local/opt/qt/ to the virtualenv so that other machines don't have to do a brew install qt? cheers, Los

— Reply to this email directly or view it on GitHub https://github.com/PySide/PySide/issues/129#issuecomment-194708407.

[mangopipeline]

So the good news is that a built a new wheel using the latest from the git repository... and everything seems to work great WITHOUT the need for the DYLD_LIBRARY_PATH

Unfortunately like Fred i'm developing apps for VFX which is standardized in PySide, plus allot of my applications are already written and working on windows....

Fred, how are handling the qt frame work? do you know of away of moving those DLL's to the virtualenv? can i move them to a folder and use some sort of environment variable to help pyside find them?

[fredrikaverpil]

Fred, how are handling the qt frame work?

I'm not. I'm currently installing it locally on each machine that needs PySide. :angry: This is of course not viable solution when you need to distribute your application to someone who does not have Qt installed locally.

You can build your own distribution of Qt from source and make the libraries load from within e.g. a virtualenv. The modifications needed to make the installation moveable is supposed to be mostly about library loading. The build system wants to add an install path and you'll need to not do that. You have to understand rpath on non-Windows platforms. However, I've not had the time to do this even if this is one of the major gripes with distributing PySide applications in my opinion. But I understand it is doable. If I ever get around doing it, I'll be sure to share this with the community. But as it looks right now, I'm hoping someone else will be doing it and sharing the build script/Qt library binaries for use in a virtualenv as I'm caught up in other projects.

It really is a major issue.

[boredstiff]

Running into this on OSX 10.11.4 and I'm running in a simple virtualenv like the others.

[garrettr]

Running into this on OSX 10.10.5. This is a problem because the PySide install docs don't work if you're using a virtualenv, which is surprising.

$ brew install qt # stable 4.8.7
$ virtualenv env # defaults to 2.7.10 on Mac 10.10.5
$ . env/bin/activate
$ pip install -U PySide # ... takes a while
$ python -c 'from PySide import QtCore'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/garrett/code/deprecated-binaryninja-python/env/lib/python2.7/site-packages/PySide/QtCore.so, 2): Library not loaded: @rpath/libpyside-python2.7.1.2.dylib
  Referenced from: /Users/garrett/code/deprecated-binaryninja-python/env/lib/python2.7/site-packages/PySide/QtCore.so
  Reason: image not found

Can confirm that the workaround from https://github.com/PySide/PySide/issues/129#issuecomment-158445160 works for me as well, but I also agreed that such a workaround should not be necessary.