search

amonakov / primus

Low-overhead client-side GPU offloading
ISC License
216 stars 20 forks source link

Getting primus to work with Intel/AMD hybrid graphics #104

Open YoRyan opened 11 years ago

YoRyan commented 11 years ago

I'm trying to hack Primus to run on my HP laptop with AMD hybrid graphics.

> lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Seymour [Radeon HD 6400M/7400M Series]

I have the two cards working together with Bumblebee and VirtualGL, but I've had no luck with Primus. I'm so close, yet so far!

Here is my modified primusrun script. Accelerating libGL is pointed to the fglrx libs, while the display libGL is pointed to the Intel/mesa libs.

#!/bin/bash

# Readback-display synchronization method
#0: no sync, 1: D lags behind one frame, 2: fully synced
# export PRIMUS_SYNC=${PRIMUS_SYNC:-0}

# Verbosity level
#0: only errors, 1: warnings (default), 2: profiling
# export PRIMUS_VERBOSE=${PRIMUS_VERBOSE:-1}

# Secondary display
# export PRIMUS_DISPLAY=${PRIMUS_DISPLAY:-:8}

# "Accelerating" libGL
# $LIB will be interpreted by the dynamic linker
export PRIMUS_libGLa=${PRIMUS_libGLa:-'/usr/$LIB/catalystpxp/fglrx/fglrx-libGL.so.1.2'}

# "Displaying" libGL
export PRIMUS_libGLd=${PRIMUS_libGLd:-'/usr/$LIB/libGL.so.1'}

# Directory containing primus libGL
PRIMUS_libGL='/usr/$LIB/primus'

# On some distributions, e.g. on Ubuntu, libnvidia-tls.so is not available
# in default search paths.  Add its path manually after the primus library
# PRIMUS_libGL=${PRIMUS_libGL}:/usr/lib/nvidia-current:/usr/lib32/nvidia-current

# Mesa drivers need a few symbols to be visible
# export PRIMUS_LOAD_GLOBAL=${PRIMUS_LOAD_GLOBAL:-'libglapi.so.0'}

# Need functions from primus libGL to take precedence
export LD_LIBRARY_PATH=${PRIMUS_libGL}${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}

# And go!
exec "$@"

I'd love to debug this and help out with development, but I've no idea where to start. Any pointers?

amonakov commented 11 years ago

If optirun glxinfo works fine and does not give (null) as the renderer string, perhaps you need the same workaround as libnvidia-tls.so issue on Ubuntu; change

# PRIMUS_libGL=${PRIMUS_libGL}:/usr/lib/nvidia-current:/usr/lib32/nvidia-current

to

PRIMUS_libGL=${PRIMUS_libGL}:'/usr/$LIB/catalystpxp/fglrx'
YoRyan commented 11 years ago

Added

PRIMUS_libGL=${PRIMUS_libGL}:'/usr/$LIB/catalystpxp'

No difference.

amonakov commented 11 years ago

Run

LD_DEBUG=libs optirun glxinfo |& tee /tmp/ld1.txt
LD_DEBUG=libs primusrun glxinfo |& tee /tmp/ld2.txt

and provide the resulting tmp/ld{1,2}.txt files (don't paste them here, use a separate pastebin).

YoRyan commented 11 years ago

Done.

amonakov commented 11 years ago

Try the following patch

diff --git a/libglfork.cpp b/libglfork.cpp
index a76227c..390d841 100644
--- a/libglfork.cpp
+++ b/libglfork.cpp
@@ -59,9 +59,9 @@ struct CapturedFns {
 #include "gl-passthru.def"
 #include "gl-needed.def"
 #undef DEF_GLX_PROTO
-  CapturedFns(const char *lib)
+  CapturedFns(const char *lib, int flag)
   {
-    handle = mdlopen(lib, RTLD_LAZY);
+    handle = mdlopen(lib, RTLD_LAZY | flag);
 #define DEF_GLX_PROTO(ret, name, args, ...) name = (ret (*) args)real_dlsym(handle, #name);
 #include "glx-reimpl.def"
 #include "glx-dpyredir.def"
@@ -241,8 +241,8 @@ static struct PrimusInfo {
     adpy(XOpenDisplay(adpy_str)),
     ddpy(XOpenDisplay(NULL)),
     needed_global(dlopen(getconf(PRIMUS_LOAD_GLOBAL), RTLD_LAZY | RTLD_GLOBAL)),
-    afns(libgla_str),
-    dfns(getconf(PRIMUS_libGLd))
+    afns(libgla_str, RTLD_GLOBAL),
+    dfns(getconf(PRIMUS_libGLd), RTLD_LOCAL)
   {
     die_if(!adpy, "failed to open secondary X display\n");
     die_if(!ddpy, "failed to open main X display\n");
YoRyan commented 11 years ago

No difference. Here is the LD_DEBUG output, if it helps. http://sprunge.us/iXhA

amonakov commented 11 years ago

Try using gdb to see what segfaults:

primusrun gdb glxinfo
r
bt
YoRyan commented 11 years ago

Unfortunately, it's not very useful...

(gdb) bt
#0  0x0000000000403a0a in ?? ()
#1  0x00000000004016e3 in ?? ()
#2  0x00007ffff74c7a15 in __libc_start_main () from /usr/lib/libc.so.6
#3  0x0000000000401ed9 in ?? ()
amonakov commented 11 years ago

I've tried it on my intel+nvidia laptop, and glxinfo appears to work fine (without direct rendering). Please produce one more LD_DEBUG log, this time with LD_DEBUG=all. Also, does the following work?

optirun -b none glxinfo -display :8
PRIMUS_DISPLAY=:0 primusrun glxinfo
YoRyan commented 11 years ago

My version of optirun lacks the bridge option (-b).

Here is the LD_DEBUG log. File size warning - 9.8 MiB.

amonakov commented 11 years ago

Build primus and glxinfo from source with debug info and find out what segfaults in glxinfo, or which call generates BadDrawable error in glxspheres.

ChristophHaag commented 11 years ago

Isn't it pretty clear where the problem is?

     15673:     calling init: /usr/lib/xorg/modules/dri/fglrx_dri.so
     15673:
     15673:     glxinfo: error: symbol lookup error: undefined symbol: AGER_Init (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: AGER_Disable (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: AGER_Term (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: __glXSetTexBufferInfo (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: __glXgetGLXPixmapInfo (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: __glXSetTexBufferInfo (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: __glXgetGLXPixmapInfo (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: AtiCallFGLComposite (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: AtiCallFGLGetDisplaySettings (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: FGL_GetRootWindows (fatal)
     15673:     glxinfo: error: symbol lookup error: undefined symbol: CleanupGLCapturer (fatal)
     15673:     find library=libGL.so.1.2 [0]; searching
     15673:      search path=/usr/lib/primus            (LD_LIBRARY_PATH)
     15673:       trying file=/usr/lib/primus/libGL.so.1.2
     15673:      search cache=/etc/ld.so.cache
     15673:      search path=/usr/lib/tls/x86_64:/usr/lib/tls:/usr/lib/x86_64:/usr/lib          (system search path)
     15673:       trying file=/usr/lib/tls/x86_64/libGL.so.1.2
     15673:       trying file=/usr/lib/tls/libGL.so.1.2
     15673:       trying file=/usr/lib/x86_64/libGL.so.1.2
     15673:       trying file=/usr/lib/libGL.so.1.2

I mean, that's a symbol from the fglrx libGL but it's opening the primus libGL...

I'm not really sure how it is supposed to work because the evnironment variables aren't really documented as far as I have seen but when I try to use primus' libGL directly with LD_PRELOAD=/usr/lib/primus/libGL.so.1 LD_LIBRARY_PATH=/usr/lib/catalystpxp/fglrx:/usr/lib/primus/ PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 I get can run glxinfo without a segfault but then again there are other symbols probably belonging to primus that are not found so glxgears etc. don't work.

That looks a little bit better I think?

     15757:     calling init: /usr/lib/xorg/modules/dri/fglrx_dri.so
     15757:
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXSetTexBufferInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXgetGLXPixmapInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXSetTexBufferInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXgetGLXPixmapInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXSetTexBufferInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXgetGLXPixmapInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXSetTexBufferInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXgetGLXPixmapInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXSetTexBufferInfo (fatal)
     15757:     glxinfo: error: symbol lookup error: undefined symbol: __glXgetGLXPixmapInfo (fatal)
     15757:     find library=libatiadlxx.so [0]; searching
     15757:      search path=/usr/lib/catalystpxp/fglrx:/usr/lib/primus         (LD_LIBRARY_PATH)
     15757:       trying file=/usr/lib/catalystpxp/fglrx/libatiadlxx.so
     15757:       trying file=/usr/lib/primus/libatiadlxx.so
     15757:      search cache=/etc/ld.so.cache
     15757:       trying file=/usr/lib/libatiadlxx.so

edit: __glXSetTexBufferInfo is from fglrx's libglx. So when calling init: /usr/lib/xorg/modules/dri/fglrx_dri.so happens it should somehow look at fglrx's libglx but I'm not sure how to tell primus that.

amonakov commented 11 years ago

@ChristophHaag, did you apply this patch? It should "fix" the first problem you've pointed out (it'll introduce other problems, but let's ignore that for now).

Regarding the second problem, have you tried adding the directory containing a libglx.so->fglrx_libglx.so (or whatever it's called) symlink to LD_LIBRARY_PATH? Thanks for looking into this.

ChristophHaag commented 11 years ago

No, I have not applied the patch. The first issue is "fixed" by preloading primus' libgl.

My /usr/lib/catalystpxp/fglrx after I played a bit around with it looks this way:

 fglrx-libGL.so.1.2*
 libGL.so -> libGL.so.1.2*
 libGL.so.1 -> libGL.so.1.2*
 libGL.so.1.2 -> fglrx-libGL.so.1.2*
 libglx.so -> /usr/lib/xorg/modules/updates/extensions/fglrx/fglrx-libglx.so*
amonakov commented 11 years ago

Well, the second set of warnings (missing __glX* symbols) is also there with VirtualGL, so presumably it's not the issue here. How does your glxinfo output look (in particular, does it report sensible renderer strings?), and how exactly does glxgears "not work"?

ChristophHaag commented 11 years ago 
$ primusrun glxinfo
ATTENTION: default value of option force_s3tc_enable overridden by environment.
name of display: :0
ATTENTION: default value of option force_s3tc_enable overridden by environment.
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: ATI
server glx version string: 1.4
server glx extensions:
    GLX_ARB_multisample, GLX_EXT_import_context, GLX_EXT_texture_from_pixmap, 
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_OML_swap_method, 
    GLX_SGIS_multisample, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, 
    GLX_SGIX_visual_select_group, GLX_SGI_make_current_read, 
    GLX_SGI_swap_control
client glx vendor string: primus
client glx version string: 1.4
client glx extensions:
    GLX_ARB_get_proc_address
GLX version: 1.4
GLX extensions:
    GLX_ARB_get_proc_address
OpenGL vendor string: (null)
OpenGL renderer string: (null)
OpenGL version string: (null)
[1]    18416 segmentation fault (core dumped)  primusrun glxinfo
$ LD_PRELOAD=/usr/lib/primus/libGL.so.1 LD_LIBRARY_PATH=/usr/lib/catalystpxp/fglrx:/usr/lib/xorg/modules/updates/extensions/fglrx/:/usr/lib/dri/:/usr/lib/primus/ PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 glxinfo
ATTENTION: default value of option force_s3tc_enable overridden by environment.
name of display: :0
ATTENTION: default value of option force_s3tc_enable overridden by environment.
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: ATI
server glx version string: 1.4
server glx extensions:
    GLX_ARB_multisample, GLX_EXT_import_context, GLX_EXT_texture_from_pixmap, 
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_OML_swap_method, 
    GLX_SGIS_multisample, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, 
    GLX_SGIX_visual_select_group, GLX_SGI_make_current_read, 
    GLX_SGI_swap_control
client glx vendor string: primus
client glx version string: 1.4
client glx extensions:
    GLX_ARB_get_proc_address
GLX version: 1.4
GLX extensions:
    GLX_ARB_get_proc_address
OpenGL vendor string: Advanced Micro Devices, Inc.
OpenGL renderer string: AMD Radeon HD 7970M
OpenGL version string: 4.3.12438 Compatibility Profile Context 13.20.5
OpenGL shading language version string: 4.30
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile
OpenGL extensions:
    GL_AMDX_debug_output, GL_AMD_blend_minmax_factor, 
    GL_AMD_conservative_depth, GL_AMD_debug_output, 
    GL_AMD_depth_clamp_separate, GL_AMD_draw_buffers_blend, 
    GL_AMD_interleaved_elements, GL_AMD_multi_draw_indirect, 
    GL_AMD_name_gen_delete, GL_AMD_performance_monitor, GL_AMD_pinned_memory, 
    GL_AMD_query_buffer_object, GL_AMD_sample_positions, 
[...] etc. etc. etc. (no segfault!)

And glxgears, well, it's just black.

blackglxgears

$ primusrun glxgears
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
ATTENTION: option value of option force_s3tc_enable ignored.
321 frames in 5.0 seconds = 64.003 FPS
301 frames in 5.0 seconds = 60.119 FPS
299 frames in 5.0 seconds = 59.723 FPS
300 frames in 5.0 seconds = 59.912 FPS
299 frames in 5.0 seconds = 59.722 FPS
299 frames in 5.0 seconds = 59.717 FPS
amonakov commented 11 years ago

Thanks. I understand this is not a priority for you, so I'm fine if the conversation ends here. I don't intend to investigate this without the hardware.

To investigate the black window issue, I would suggest to first see if fglrx-to-fglrx offloading via primus works (so that Mesa libraries are not loaded at all). You can connect to the invisible secondary X server with Xpra to do that.

YoRyan commented 11 years ago

Apologize for my absence - had to be out of town for several days.

Here is the gdb output of glxinfo 8.1.0 with debugging symbols: http://sprunge.us/IbIY

It seems the problem is with the GL version, of all things!

Not sure how to debug the BadDrawable in glxspheres, any hints for a noob?

amonakov commented 11 years ago

glxinfo tries to dereference the version string, which is null, so it segfaults.

For BadDrawable, for an inexact answer you can put a breakpoint on exit in gdb

ChristophHaag commented 11 years ago

@amonakov

How would I set up fglrx-to-fglrx offloading via primus and how does connecting with Xpra work? I mean xpra attach :8 doesn't work for normal xservers does it? connection failed: [Errno 2] No such file or directory

amonakov commented 11 years ago

How would I set up fglrx-to-fglrx offloading via primus

You need to set both PRIMUS_libGLa and PRIMUS_libGLd to fglrx's libGL, and DISPLAY to :8 (like PRIMUS_DISPLAY). There's also PRIMUS_LOAD_GLOBAL that by default points to Mesa's libglapi, you should check if setting it to something unrelated like /lib/libdl.so.2 makes things better (there's a symbol clash with fglrx's libGL, but it did not look like a problem to me because the library has symboling binding, and other fgrlx libraries didn't seem to reference those symbols).

I mean xpra attach :8 doesn't work for normal xservers does it?

xpra start :8 --use-display
xpra attach :8
ChristophHaag commented 11 years ago

Ok, thanks. I did

DISPLAY=:8 LD_PRELOAD=/usr/lib/primus/libGL.so.1 LD_LIBRARY_PATH=/usr/lib/catalystpxp/fglrx PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 glxgears -info
GL_RENDERER   = AMD Radeon HD 7970M
GL_VERSION    = 4.3.12438 Compatibility Profile Context 13.20.5
GL_VENDOR     = ATI Technologies Inc.
GL_EXTENSIONS = etc. etc. etc.

and then xpra start :8 --use-display and then xpra attach :8. With this I see glxgears.

With primusrun the PRIMUS_LOAD_GLOBAL did get in the way so I had to use

. primusrun like that works with xpra:

#!/bin/bash
export DISPLAY=:8
export PRIMUS_libGLa='/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2'
export PRIMUS_libGLd='/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2'
PRIMUS_libGL='/usr/lib/primus'
export PRIMUS_LOAD_GLOBAL="/lib/libdl.so.2"
export LD_LIBRARY_PATH="$PRIMUS_libGL:/usr/lib/catalystpxp/fglrx"
exec "$@"

But when I instead use PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 then I only see a black window. But maybe it just doesn't work the way I did it because I get messages like libGL: screen 0 does not appear to be DRI2 capable

amonakov commented 11 years ago

Is that still without the above patch? How did primusrun fail before adjusting LOAD_GLOBAL?

ChristophHaag commented 11 years ago

I am using the patch now.

Sorry, I was still a bit tired when I wrote the above.

Now correctly.

  1. With nothing useful in PRIMUS_LOAD_GLOBAL: LIBGL_DEBUG=verbose LD_PRELOAD=/usr/lib/primus/libGL.so.1 LD_LIBRARY_PATH=/usr/lib/primus/:/usr/lib/catalystpxp/fglrx PRIMUS_LOAD_GLOBAL=/lib/libdl.so.2 PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 glxgears -info

I get libGL error: dlopen /usr/lib/xorg/modules/dri/i965_dri.so failed (/usr/lib/libdricore9.3.0-devel.so.1: undefined symbol: _glapi_tls_Context) but I get the correct version string:

GL_RENDERER   = AMD Radeon HD 7970M
GL_VERSION    = 4.3.12438 Compatibility Profile Context 13.20.5
GL_VENDOR     = ATI Technologies Inc.
  1. Load libglapi in PRIMUS_LOAD_GLOBAL: LIBGL_DEBUG=verbose LD_PRELOAD=/usr/lib/primus/libGL.so.1 LD_LIBRARY_PATH=/usr/lib/primus/:/usr/lib/catalystpxp/fglrx PRIMUS_LOAD_GLOBAL=/usr/lib/libglapi.so.0.0.0 PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 glxgears -info

With mesa's libglapi in PRIMUS_LOAD_GLOBAL it seems to load the i965 driver correctly, but the I get the wrong version string as for fglrx:

GL_RENDERER   = (null)
GL_VERSION    = (null)
GL_VENDOR     = (null)
GL_EXTENSIONS = (null)

It doesn't seem to like mesa's libglapi.

Now for my 3. try, additionally preloading fglrx's libGL: LIBGL_DEBUG=verbose LD_PRELOAD=/usr/lib/primus/libGL.so.1:/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 LD_LIBRARY_PATH=/usr/lib/primus/:/usr/lib/catalystpxp/fglrx PRIMUS_LOAD_GLOBAL=/usr/lib/libglapi.so.0.0.0 PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 glxgears -info

It seems to load the i965 driver correctly, I get the correct OpenGL version string from fglrx, but the window content of glxgears is gray and it spams function is no-op to the console.

I'm not really sure if I'm doing it wrong.

edit: I lost PRIMUS_libGL='/usr/lib/primus' but it doesn't make a difference as PRIMUS_libGL='/usr/lib/primus' and LD_PRELOAD=/usr/lib/primus/libGL.so.1 seem to have the same effect. Only when I remove primus' libGL from LD_RELOAD and have PRIMUS_libGL instead I get a segfault: LIBGL_DEBUG=verbose PRIMUS_libGL='/usr/lib/primus' LD_PRELOAD=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 LD_LIBRARY_PATH=/usr/lib/primus/:/usr/lib/catalystpxp/fglrx PRIMUS_LOAD_GLOBAL=/usr/lib/libglapi.so.0.0.0 PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 gdb --args glxgears -info

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7e71c74 in ?? () from /usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2
(gdb) bt
#0  0x00007ffff7e71c74 in ?? () from /usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2
#1  0x00007ffff7e5c067 in ?? () from /usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2
#2  0x00007ffff7e5c375 in glXMakeCurrent () from /usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2
#3  0x0000000000401ad2 in ?? ()
#4  0x00007ffff73aca15 in __libc_start_main () from /usr/lib/libc.so.6
#5  0x00000000004023fd in ?? ()
amonakov commented 11 years ago

OK, thanks. The reason it doesn't work as intended is that the glapi symbol clash is, in fact, a problem, and fglrx_dri.so ends up using Mesa's glapi functions instead of fglrx's. I don't see a way to fix it except of an LD_AUDIT helper.

amonakov commented 11 years ago

No.

ChristophHaag commented 11 years ago

Is anyone working on this? I'd be already happy with an ugly workaround like a library that provides a dlopen() with hardcoded behavior or so.

amonakov commented 11 years ago

Apparently nobody is working on this. Why are you interested in getting the fglrx/primus combo working, is the official approach that involves restarting the X server not satisfactory?

ChristophHaag commented 11 years ago

Why? Have you tried using that official approach? I consider having to restart X a very bad solution. Then it's not possible to run one X with intel and one with amd so that you don't have to kill all your gui programs every time you want to render something with the dedicated gpu. Then switching multiple times from amd to intel and back often doesn't work and you even have to reboot to get it working again. And with more than one monitor it goes from severe rendering problems to rendering complete garbage.

Up to the 13.8 beta it looks like this

Maybe it's just with the 7970M and not with older gpus...

Anyway, bumblebee with virtualgl works okay, but the performance... I don't really know whether it is good or could be much better.

quantax0 commented 11 years ago

@ChristophHaag An "ugly workaround" utilizing the 'official approach' would be to start up X with Intel, then switch to the AMD gpu (I use @YoRyan 's ati script, though Vi0l0's pxp script probably would work fine too), then ctrl-alt-f2 to the second tty and start an X-server there. I can then start up any number of games and run them in that second X server, without having to kill the original Intel server (and all my gui apps). However, I run into some issues, namely:

Right now it's to a point where it works, at least in the loosest of terms, and I haven't been quite as compelled to spend as much time in Windows, and that's always good. Getting primus to run with AMD/Intel would be a huge perk.

YoRyan commented 11 years ago

Dual monitors is also my motivation for using Bumblebee with fglrx + Intel. I don't know if it's just the way the HDMI port is wired on this laptop, but the screen is absolutely borked when I turn on the second monitor with the AMD GPU running, just like in @ChristophHaag 's screenshot.

By the way, I have just updated my ATI scripts to no longer depend on Vi0l0's switching system. This generally makes everything more reliable.

@quantax0 , I'm guessing your solution fails when you try to launch a new GLX app on the Intel X server, because Vi0l0's (and my old) scripts alter the global libGL library.

...And finally, abysmal fglrx performance aside, I suspect that fglrx is also causing suspend-to-RAM issues on my laptop. I think this is the last straw for this troublesome GPU - I might just disable it on bootup and do all my gaming under Losedows. EDIT: Suspend bug unrelated to fglrx, fixed in kernel 3.11.

amonakov commented 11 years ago

Well. Try this: binary patching fglrx's libraries to sidestep the symbol clash issue:

#!/bin/bash
for tfile in `find /usr -name fglrx-libGL.so.1.2 -o -name fglrx_dri.so`; do
  echo "Backing up $tfile -> ${tfile}.bak"
  cp $tfile ${tfile}.bak
  echo "Patching   $tfile"
  sed -i s/_glapi_/_glaoy_/g $tfile
done

(the _glapi_ prefix is adjusted in a way that keeps ELF symbol hashes unchanged) (you'll have to run the above with superuser privileges)

ChristophHaag commented 11 years ago

Yes, that's good enough for now.

The minimal set of variables I seem to need is LD_LIBRARY_PATH=/usr/lib/primus PRIMUS_libGL=/usr/lib/primus PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 glxgears -info

That prints

GL_RENDERER   = AMD Radeon HD 7970M
GL_VERSION    = 4.3.12441 Compatibility Profile Context 13.20.11
GL_VENDOR     = ATI Technologies Inc.
GL_EXTENSIONS = GL_AMDX_debug_output ... and all the other extensions

and it actually displays glxgears!

I don't even need PRIMUS_LOAD_GLOBAL=/usr/lib/libglapi.so.0.0.0 right now.

Without LD_LIBRARY_PATH=/usr/lib/primus (PRIMUS_libGL=/usr/lib/primus PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 glxgears -info) it runs with intel...

With 32 bit I didn't have so much luck yet, but for 64 bit it seems to work.

Performance is by far not what I hoped. :( Tried xonotic-glx and with intel graphics it was more fluent.

Unigine Valley has a benchmark so I tried it. Unigine says on both, primus and optirun/virtualgl GPU model: Unknown GPU (256MB) x1 but it definitely uses the amd gpu because it doesn't even start (out of the box) on mesa drivers, at least intel. (It should say GPU model: AMD Radeon HD 7970M (2048MB) x1. I don't know if unigine valley cuts something down on itself because of that...) primus:

FPS:    
13.6
Score:  
571
Min FPS:    
9.8
Max FPS:    
20.3

optirun/virtualgl:

FPS:    
25.1
Score:  
1049
Min FPS:    
15.2
Max FPS:    
36.6

With both the gpu load hovers somewhere around 50%-60% and only sometimes goes to ~70%

Not sure if the AMD driver is just bad or primus/virtualgl are such bottlenecks.

I had the thought that I can just do DISPLAY=:8 LD_LIBRARY_PATH=/usr/lib/catalystpxp/fglrx/ unigine-valley and use vnc to start the benchmark so I did just that. "Natively" it runs with ~70-90% gpu load

FPS:    
36.3
Score:  
1517
Min FPS:    
18.6
Max FPS:    
56.8
amonakov commented 11 years ago

Can you show logs with vblank_mode=0 PRIMUS_VERBOSE=2 for both glxgears and Valley?

ChristophHaag commented 11 years ago 
vblank_mode=0 PRIMUS_VERBOSE=2 LD_LIBRARY_PATH=/usr/lib/primus PRIMUS_libGL=/usr/lib/primus PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 glxgears
ATTENTION: default value of option vblank_mode overridden by environment.
ATTENTION: default value of option vblank_mode overridden by environment.
ATTENTION: default value of option vblank_mode overridden by environment.
ATTENTION: default value of option vblank_mode overridden by environment.
ATTENTION: default value of option vblank_mode overridden by environment.
ATTENTION: default value of option vblank_mode overridden by environment.
ATTENTION: option value of option vblank_mode ignored.
9285 frames in 5.0 seconds = 1856.968 FPS
primus: profiling: display: 1860.3 fps, 12.9% wait, 51.0% upload, 36.0% draw+swap
primus: profiling: readback: 1860.5 fps, 20.0% app, 48.5% map, 31.5% wait
9514 frames in 5.0 seconds = 1902.749 FPS
primus: profiling: display: 1903.2 fps, 10.9% wait, 52.3% upload, 36.8% draw+swap
primus: profiling: readback: 1903.2 fps, 19.0% app, 48.9% map, 32.1% wait
9548 frames in 5.0 seconds = 1909.527 FPS
primus: profiling: display: 1910.4 fps, 10.8% wait, 52.2% upload, 37.0% draw+swap
primus: profiling: readback: 1910.4 fps, 18.9% app, 49.0% map, 32.1% wait
9535 frames in 5.0 seconds = 1906.929 FPS
primus: profiling: readback: 1906.8 fps, 19.0% app, 49.0% map, 32.0% wait
primus: profiling: display: 1906.9 fps, 11.1% wait, 52.2% upload, 36.7% draw+swap
9520 frames in 5.0 seconds = 1903.106 FPS
primus: profiling: readback: 1904.0 fps, 19.0% app, 49.0% map, 32.0% wait
primus: profiling: display: 1904.0 fps, 11.2% wait, 52.2% upload, 36.6% draw+swap
9495 frames in 5.0 seconds = 1898.883 FPS
primus: profiling: display: 1896.5 fps, 11.2% wait, 52.1% upload, 36.8% draw+swap
primus: profiling: readback: 1896.7 fps, 19.1% app, 48.9% map, 32.0% wait
Unigine Valley Benchmark 1.0 (1.0)Unigine~# world_load valley/valley
Loading "valley/valley.cpp" 170ms
Loading "valley/valley.mat" 72 materials 1379ms
Loading "valley/sound/sound.prop" 142 properties 1ms
Loading "valley/valley.world" 3285ms
primus: profiling: readback: 11.1 fps, 93.1% app, 3.3% map, 3.6% wait
primus: profiling: display: 11.1 fps, 92.1% wait, 7.4% upload, 0.4% draw+swap
primus: profiling: readback: 4.3 fps, 97.8% app, 2.1% map, 0.1% wait
primus: profiling: display: 4.3 fps, 98.2% wait, 1.7% upload, 0.1% draw+swap
Benchmark running
primus: profiling: readback: 14.2 fps, 93.4% app, 6.1% map, 0.5% wait
primus: profiling: display: 14.2 fps, 93.3% wait, 6.3% upload, 0.3% draw+swap
primus: profiling: readback: 15.2 fps, 93.6% app, 5.9% map, 0.4% wait
primus: profiling: display: 15.2 fps, 93.7% wait, 6.0% upload, 0.4% draw+swap
primus: profiling: readback: 17.3 fps, 92.4% app, 6.7% map, 0.9% wait
primus: profiling: display: 17.3 fps, 92.3% wait, 7.3% upload, 0.4% draw+swap
primus: profiling: readback: 15.5 fps, 96.1% app, 3.2% map, 0.7% wait
primus: profiling: display: 15.5 fps, 92.4% wait, 7.2% upload, 0.4% draw+swap
primus: profiling: readback: 14.4 fps, 96.3% app, 3.1% map, 0.6% wait
primus: profiling: display: 14.4 fps, 93.4% wait, 6.2% upload, 0.3% draw+swap
primus: profiling: readback: 14.2 fps, 95.7% app, 3.7% map, 0.6% wait
primus: profiling: display: 14.2 fps, 93.7% wait, 5.9% upload, 0.3% draw+swap
primus: profiling: readback: 14.2 fps, 95.2% app, 4.2% map, 0.6% wait
primus: profiling: display: 14.2 fps, 93.2% wait, 6.4% upload, 0.4% draw+swap

With or without compositing doesn't make a big difference... maybe a few percent more...

# after disabling kwin compositing while unigine valley was running
primus: profiling: readback: 14.2 fps, 94.5% app, 4.9% map, 0.6% wait
primus: profiling: display: 14.2 fps, 94.1% wait, 5.5% upload, 0.4% draw+swap
primus: profiling: readback: 15.2 fps, 94.2% app, 5.3% map, 0.5% wait
primus: profiling: display: 15.2 fps, 93.7% wait, 6.0% upload, 0.3% draw+swap
primus: profiling: readback: 15.4 fps, 95.4% app, 4.0% map, 0.6% wait
primus: profiling: display: 15.4 fps, 93.8% wait, 5.8% upload, 0.4% draw+swap
primus: profiling: readback: 15.2 fps, 96.4% app, 3.2% map, 0.4% wait
primus: profiling: display: 15.2 fps, 94.0% wait, 5.7% upload, 0.4% draw+swap
primus: profiling: readback: 14.3 fps, 96.8% app, 2.9% map, 0.3% wait
primus: profiling: display: 14.3 fps, 94.3% wait, 5.4% upload, 0.3% draw+swap
amonakov commented 11 years ago

Does PRIMUS_SYNC=1 improve things?

ChristophHaag commented 11 years ago

glxgears: PRIMUS_SYNC=1 -> ~800 fps PRIMUS_SYNC=0 -> ~1900 fps

In unigine valley there is not a real difference.

Thanks for your engagement, by the way. Pretty awesome.

edit: Removed my stupid comment because I did forget to apply the patch after updating. Sorry!

amonakov commented 11 years ago

Did you reapply the patch after updating? You must have had it before.

ChristophHaag commented 11 years ago

No, i didn't and with the patch for 32 bit it works there too.

So I tested a bit more and maybe unigine valley is an unfair benchmark.

Another one that also uses not ancient opengl, unfortunately closed source too: http://www.geeks3d.com/gputest/download/ I ran the windowed scripts, no compositing, vblank_mode=0, gpu load as reported by aticonfig --odgc. Not very scientific, just picking a representative looking fps to get a superficial comparison.

furmark: virtualgl: 63 fps, 84% gpu load primus: 71 fps, 95% gpu load

gimark: virtualgl: 77fps, 78% gpu load primus: 91 fps, 87% gpu load

piano: virtualgl: 17 fps, 96% gpu load primus: 17 fps, 98% gpu load

volplosion: virtualgl: 44 fps, 90% gpu load primus: 47 fps, 96% gpu load

tessmark is strange, apparently no gpu load with virtualgl and the gpu clocks down. Only ~70fps, with primus 88% gpu load and 232 fps. Probably broken with optirun.

So unigine-valley is still much slower than with virtualgl, but at least it seems to be an application specific problem and not some horrible inefficiency in primus. :)

amonakov commented 11 years ago

Can you add "native" and "official offloading" fps figures for comparison? Among the above, gimark would be most interesting.

What about games, as I understand, fluidity on Xonotic is not good enough (is it better/worse with VirtualGL?).

ChristophHaag commented 11 years ago

primusrun with PRIMUS_SYNC=0 vblank_mode=0 and everything without compositing.

method  resolution      avg fps gimark score
primus  1920x1080       75      4537
vgl     1920x1080       56      3364
native  1920x1080       89      5380
primus  1024x640        81      4898
vgl     1024x640        77      4674
native  1024x640        89      5375

The official I'll maybe do later, if it even works.

The gputest benchmarks all run better with primus. I have tried a few games like trine2, natural selection 2, euro truck simulator 2, ... With PRIMUS_SYNC=0 they are very prone to just freeze after a few seconds to a few minutes, even without compositing and the short time it works the performance is not very good. With PRIMUS_SYNC=1 the performance is pretty bad... Maybe one of the games that run bad has detailed profiling...

f-r-i-t-z commented 11 years ago

ChristophHaag, let me say I want to try this line but with radeon not fglrx. Can yu give me a glimpse?


LD_LIBRARY_PATH=/usr/lib/primus PRIMUS_libGL=/usr/lib/primus PRIMUS_libGLa=/usr/lib/catalystpxp/fglrx/fglrx-libGL.so.1.2 PRIMUS_libGLd=/usr/lib/libGL.so.1.2.0 glxgears -info
quantax0 commented 11 years ago

@f-r-i-t-z I'm not familiar with the radeon module, but nothing primus does should have any problem with it. You'll want to change change references to fglrx to radeon, ie, changing libGLa to point to the libGL for radeon (mesa? i'm not too familiar with the open source module).

Keep in mind that you'll likely run into other issues with the radeon module if you're running a 7xxxM card, because i'm not sure that the radeon module will actually work with that series.

quantax0 commented 11 years ago

@f-r-i-t-z if you want to use radeon you're better off using PRIME offloading, as that is a cleaner solution and will provide you with about as good performance as you can expect given the state of the open source drivers.

f-r-i-t-z commented 11 years ago

@quantax0 Yeap... this is what I'm using now, dri_prime. but yu know. the curiosity of use bumblebee+primus was bigger and sometimes is good to find out how everything is going/advantages. No, I'm not using 7xxxM series, i'm using 68xxM series. WORST ( nor the catalyst driver like this one).. Well. thank you by your answer :-)