Maximum number of clients reached

[JHagemeister]

When running an auto-test-suite (Catch2 + Qt) which includes subsequent creation of many application windows triggers the following issue

"Maximum number of clients reached[VGL] ERROR: in VirtualWin--[VGL] 77: Could not clone X display connection"

This happens because FBXFrames create their own display connection when instantiated in X11Trans::getFrame. The display connection is closed in the destructor of FBXFrame which happens at the very end of the test suite when a large amount of FBXFrames have piled up.

Observed call stack of FBXFrame creation: X11Trans::getFrame VirtualWin::sendx11 VirtualWin::readback glxSwapBuffers

The issue is fixed for me if I change the instantiation of FBXFrame in x11Trans::getFrame to reuse existing display connection

frames[index] = new FBXFrame(dpy, win, NULL, true);

[dcommander]

There are several facets to this issue.

1. FBXFrame has to use a separate X display connection for thread safety, since VirtualGL doesn't know when it is safe to draw images using the existing display connection. Reusing the existing display connection may accidentally work for your application, but it definitely would not work in the general case.
2. X11Trans has to use a pool of three FBXFrame instances in order for frame spoiling to work correctly.
3. If VirtualGL detects that the 3D application is not monitoring window resize events, then a separate display connection will be created in VirtualWin for the purposes of monitoring those events. (This is necessary so VirtualGL can resize the Pbuffer associated with an OpenGL window if the window size changes.)

This is all triggered by the creation of a VirtualWin instance for an X window, which will only occur if the window is made current using glXMake*Current().

Since X clients can have as many as 256 simultaneous display connections, your application must be creating at least 64 X windows and attempting to use those windows simultaneously with OpenGL. That is, as you can imagine, not a common scenario. (The VirtualGL faker unit test tests 30 simultaneous windows.)

There are several things that could be done to reduce the number of X display connections that VirtualGL creates:

1. Modify the application so that it monitors X window resize events, or if the application does not ever resize its windows, run it with vglrun +wm. That will prevent VirtualWin from cloning the display connection for event monitoring purposes, and it will thus reduce the number of X display connections per window from 4 to 3.
2. VirtualGL could potentially reduce its pool of FBXFrame instances in X11Trans from three to one if VGL_SYNC/vglrun +sync is used, but note the limitations of that mode. It may be OK for an automated test application, but it is definitely not suitable for all applications. If the test application is performance-critical, then it may not even be suitable for that.
3. It may be possible for X11Trans to reduce the number of FBXFrame instances in its pool from three to two if frame spoiling is disabled, but this would be a pain to implement and test, since the frame spoiling setting can be changed dynamically.

[JHagemeister]

Thank you very much for taking the time for this elaborate answer!

Based on your reply, it seems we might rather have a bug in our code. I will continue trouble shooting.

[JHagemeister]

After further investigation, we were able to come up with a minimal example of this issue that purely makes use of Qt and VirtualGL code. We would very much appreciate if you could have another look as to whether this might actually be a VirtualGL bug.

(Note that it is a txt file as I was not allowed to attach a cc file :)) vgl-max-clients-reached-bug.txt

[tsondergaard]

I tracked a bunch of VirtualWin constructions and destructions:

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x5555559925c0 vglfaker::VirtualWin
        win 48234505    Window
        xwa @0x7fffffffc630 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1  vglfaker::VirtualWin::VirtualWin                VirtualWin.cpp                 51    0x7ffff7f3b359
2  vglfaker::WindowHash::initVW                    WindowHash.h                   69    0x7ffff7f10c0b
3  glXMakeCurrent                                  faker-glx.cpp                  1592  0x7ffff7f05746
4  QGLXContext::init                               qglxintegration.cpp            415   0x7fffe4f7998b
5  QXcbGlxIntegration::createPlatformOpenGLContext qxcbglxintegration.cpp         194   0x7fffe4f77261
6  QOpenGLContext::create                          qopenglcontext.cpp             612   0x7ffff7314f76
7  QWidgetPrivate::shareContext                    qwidget.cpp                    12006 0x7ffff79ee942
8  QOpenGLWidgetPrivate::initialize                qopenglwidget.cpp              796   0x7ffff7a1f03d
9  QOpenGLWidget::resizeEvent                      qopenglwidget.cpp              1286  0x7ffff7a1f19b
10 QWidget::event                                  qwidget.cpp                    8809  0x7ffff7a00c65
11 QApplicationPrivate::notify_helper              qapplication.cpp               3632  0x7ffff79bfa02
12 QCoreApplication::notifyInternal2               qcoreapplication.cpp           1063  0x7ffff6ecf8d1
13 QWidgetPrivate::sendPendingMoveAndResizeEvents  qwidget.cpp                    7711  0x7ffff79f8d83
14 QWidgetPrivate::show_helper                     qwidget.cpp                    7762  0x7ffff79fd688
15 QWidgetPrivate::setVisible                      qwidget.cpp                    8111  0x7ffff7a0011d
16 main                                            vgl-max-clients-reached-bug.cc 30    0x555555556299

Locals
        this    @0x5555559925c0 vglfaker::VirtualWin

1  vglfaker::VirtualWin::~VirtualWin                                        VirtualWin.cpp                 90    0x7ffff7f3b921
2  vglfaker::WindowHash::detach                                             WindowHash.h                   117   0x7ffff7ee5aca
3  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::killEntry Hash.h                         137   0x7ffff7ee6515
4  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::remove    Hash.h                         105   0x7ffff7f11d60
5  vglfaker::WindowHash::remove                                             WindowHash.h                   86    0x7ffff7f10cee
6  DeleteWindow                                                             faker-x11.cpp                  251   0x7ffff7f16b2f
7  XDestroyWindow                                                           faker-x11.cpp                  296   0x7ffff7f17b89
8  QXcbGlxIntegration::createPlatformOpenGLContext                          qxcbglxintegration.cpp         194   0x7fffe4f77261
9  QOpenGLContext::create                                                   qopenglcontext.cpp             612   0x7ffff7314f76
10 QWidgetPrivate::shareContext                                             qwidget.cpp                    12006 0x7ffff79ee942
11 QOpenGLWidgetPrivate::initialize                                         qopenglwidget.cpp              796   0x7ffff7a1f03d
12 QOpenGLWidget::resizeEvent                                               qopenglwidget.cpp              1286  0x7ffff7a1f19b
13 QWidget::event                                                           qwidget.cpp                    8809  0x7ffff7a00c65
14 QApplicationPrivate::notify_helper                                       qapplication.cpp               3632  0x7ffff79bfa02
15 QCoreApplication::notifyInternal2                                        qcoreapplication.cpp           1063  0x7ffff6ecf8d1
16 QWidgetPrivate::sendPendingMoveAndResizeEvents                           qwidget.cpp                    7711  0x7ffff79f8d83
17 QWidgetPrivate::show_helper                                              qwidget.cpp                    7762  0x7ffff79fd688
18 QWidgetPrivate::setVisible                                               qwidget.cpp                    8111  0x7ffff7a0011d
19 main                                                                     vgl-max-clients-reached-bug.cc 30    0x555555556299

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x555555b89810 vglfaker::VirtualWin
        win 48234507    Window
        xwa @0x7fffffffc670 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1  vglfaker::VirtualWin::VirtualWin                VirtualWin.cpp                 51   0x7ffff7f3b359
2  vglfaker::WindowHash::initVW                    WindowHash.h                   69   0x7ffff7f10c0b
3  glXMakeCurrent                                  faker-glx.cpp                  1592 0x7ffff7f05746
4  QGLXContext::init                               qglxintegration.cpp            415  0x7fffe4f7998b
5  QXcbGlxIntegration::createPlatformOpenGLContext qxcbglxintegration.cpp         194  0x7fffe4f77261
6  QOpenGLContext::create                          qopenglcontext.cpp             612  0x7ffff7314f76
7  QOpenGLWidgetPrivate::initialize                qopenglwidget.cpp              813  0x7ffff7a1edd7
8  QOpenGLWidget::resizeEvent                      qopenglwidget.cpp              1286 0x7ffff7a1f19b
9  QWidget::event                                  qwidget.cpp                    8809 0x7ffff7a00c65
10 QApplicationPrivate::notify_helper              qapplication.cpp               3632 0x7ffff79bfa02
11 QCoreApplication::notifyInternal2               qcoreapplication.cpp           1063 0x7ffff6ecf8d1
12 QWidgetPrivate::sendPendingMoveAndResizeEvents  qwidget.cpp                    7711 0x7ffff79f8d83
13 QWidgetPrivate::show_helper                     qwidget.cpp                    7762 0x7ffff79fd688
14 QWidgetPrivate::setVisible                      qwidget.cpp                    8111 0x7ffff7a0011d
15 main                                            vgl-max-clients-reached-bug.cc 30   0x555555556299

Locals
        this    @0x555555b89810 vglfaker::VirtualWin

1  vglfaker::VirtualWin::~VirtualWin                                        VirtualWin.cpp                 90   0x7ffff7f3b921
2  vglfaker::WindowHash::detach                                             WindowHash.h                   117  0x7ffff7ee5aca
3  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::killEntry Hash.h                         137  0x7ffff7ee6515
4  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::remove    Hash.h                         105  0x7ffff7f11d60
5  vglfaker::WindowHash::remove                                             WindowHash.h                   86   0x7ffff7f10cee
6  DeleteWindow                                                             faker-x11.cpp                  251  0x7ffff7f16b2f
7  XDestroyWindow                                                           faker-x11.cpp                  296  0x7ffff7f17b89
8  QXcbGlxIntegration::createPlatformOpenGLContext                          qxcbglxintegration.cpp         194  0x7fffe4f77261
9  QOpenGLContext::create                                                   qopenglcontext.cpp             612  0x7ffff7314f76
10 QOpenGLWidgetPrivate::initialize                                         qopenglwidget.cpp              813  0x7ffff7a1edd7
11 QOpenGLWidget::resizeEvent                                               qopenglwidget.cpp              1286 0x7ffff7a1f19b
12 QWidget::event                                                           qwidget.cpp                    8809 0x7ffff7a00c65
13 QApplicationPrivate::notify_helper                                       qapplication.cpp               3632 0x7ffff79bfa02
14 QCoreApplication::notifyInternal2                                        qcoreapplication.cpp           1063 0x7ffff6ecf8d1
15 QWidgetPrivate::sendPendingMoveAndResizeEvents                           qwidget.cpp                    7711 0x7ffff79f8d83
16 QWidgetPrivate::show_helper                                              qwidget.cpp                    7762 0x7ffff79fd688
17 QWidgetPrivate::setVisible                                               qwidget.cpp                    8111 0x7ffff7a0011d
18 main                                                                     vgl-max-clients-reached-bug.cc 30   0x555555556299

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x555555f38720 vglfaker::VirtualWin
        win 48234515    Window
        xwa @0x7fffffffc100 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1   vglfaker::VirtualWin::VirtualWin                VirtualWin.cpp             51   0x7ffff7f3b359
2   vglfaker::WindowHash::initVW                    WindowHash.h               69   0x7ffff7f10c0b
3   glXMakeCurrent                                  faker-glx.cpp              1592 0x7ffff7f05746
4   QGLXContext::init                               qglxintegration.cpp        415  0x7fffe4f7998b
5   QXcbGlxIntegration::createPlatformOpenGLContext qxcbglxintegration.cpp     194  0x7fffe4f77261
6   QOpenGLContext::create                          qopenglcontext.cpp         612  0x7ffff7314f76
7   QPlatformBackingStore::composeAndFlush          qplatformbackingstore.cpp  335  0x7ffff7620e57
8   QXcbBackingStore::composeAndFlush               qxcbbackingstore.cpp       900  0x7fffe64189db
9   QXcbBackingStore::composeAndFlush               qxcbbackingstore.cpp       891  0x7fffe64189db
10  QWidgetRepaintManager::flush                    qwidgetrepaintmanager.cpp  1179 0x7ffff79ccc80
11  QWidgetRepaintManager::flush                    qwidgetrepaintmanager.cpp  1082 0x7ffff79ce932
12  QWidgetRepaintManager::paintAndFlush            qwidgetrepaintmanager.cpp  1014 0x7ffff79d09d4
13  QWidgetRepaintManager::sync                     qwidgetrepaintmanager.cpp  743  0x7ffff79d11d7
14  QWidgetWindow::event                            qwidgetwindow.cpp          347  0x7ffff7a1cb50
15  QApplicationPrivate::notify_helper              qapplication.cpp           3632 0x7ffff79bfa02
16  QCoreApplication::notifyInternal2               qcoreapplication.cpp       1063 0x7ffff6ecf8d1
17  QGuiApplicationPrivate::processExposeEvent      qguiapplication.cpp        3258 0x7ffff72d232f
18  QWindowSystemInterface::sendWindowSystemEvents  qwindowsysteminterface.cpp 1169 0x7ffff72a6b7c
19  QXcbUnixEventDispatcher::processEvents          qxcbeventdispatcher.cpp    63   0x7fffe642227f
20  QEventLoop::exec                                qflags.h                   69   0x7ffff6ece3eb
... <More>

Locals
        this    @0x555555f38720 vglfaker::VirtualWin

1   vglfaker::VirtualWin::~VirtualWin                                        VirtualWin.cpp            90   0x7ffff7f3b921
2   vglfaker::WindowHash::detach                                             WindowHash.h              117  0x7ffff7ee5aca
3   vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::killEntry Hash.h                    137  0x7ffff7ee6515
4   vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::remove    Hash.h                    105  0x7ffff7f11d60
5   vglfaker::WindowHash::remove                                             WindowHash.h              86   0x7ffff7f10cee
6   DeleteWindow                                                             faker-x11.cpp             251  0x7ffff7f16b2f
7   XDestroyWindow                                                           faker-x11.cpp             296  0x7ffff7f17b89
8   QXcbGlxIntegration::createPlatformOpenGLContext                          qxcbglxintegration.cpp    194  0x7fffe4f77261
9   QOpenGLContext::create                                                   qopenglcontext.cpp        612  0x7ffff7314f76
10  QPlatformBackingStore::composeAndFlush                                   qplatformbackingstore.cpp 335  0x7ffff7620e57
11  QXcbBackingStore::composeAndFlush                                        qxcbbackingstore.cpp      900  0x7fffe64189db
12  QXcbBackingStore::composeAndFlush                                        qxcbbackingstore.cpp      891  0x7fffe64189db
13  QWidgetRepaintManager::flush                                             qwidgetrepaintmanager.cpp 1179 0x7ffff79ccc80
14  QWidgetRepaintManager::flush                                             qwidgetrepaintmanager.cpp 1082 0x7ffff79ce932
15  QWidgetRepaintManager::paintAndFlush                                     qwidgetrepaintmanager.cpp 1014 0x7ffff79d09d4
16  QWidgetRepaintManager::sync                                              qwidgetrepaintmanager.cpp 743  0x7ffff79d11d7
17  QWidgetWindow::event                                                     qwidgetwindow.cpp         347  0x7ffff7a1cb50
18  QApplicationPrivate::notify_helper                                       qapplication.cpp          3632 0x7ffff79bfa02
19  QCoreApplication::notifyInternal2                                        qcoreapplication.cpp      1063 0x7ffff6ecf8d1
20  QGuiApplicationPrivate::processExposeEvent                               qguiapplication.cpp       3258 0x7ffff72d232f
... <More>

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x555555f38720 vglfaker::VirtualWin
        win 48234502    Window
        xwa @0x7fffffffc1c0 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1   vglfaker::VirtualWin::VirtualWin               VirtualWin.cpp                 51   0x7ffff7f3b359
2   vglfaker::WindowHash::initVW                   WindowHash.h                   69   0x7ffff7f10c0b
3   glXMakeCurrent                                 faker-glx.cpp                  1604 0x7ffff7f05802
4   QGLXContext::makeCurrent                       qglxintegration.cpp            571  0x7fffe4f78ec4
5   QOpenGLContext::makeCurrent                    qopenglcontext.cpp             992  0x7ffff73123cc
6   QPlatformBackingStore::composeAndFlush         qscopedpointer.h               116  0x7ffff761ffed
7   QXcbBackingStore::composeAndFlush              qxcbbackingstore.cpp           900  0x7fffe64189db
8   QXcbBackingStore::composeAndFlush              qxcbbackingstore.cpp           891  0x7fffe64189db
9   QWidgetRepaintManager::flush                   qwidgetrepaintmanager.cpp      1179 0x7ffff79ccc80
10  QWidgetRepaintManager::flush                   qwidgetrepaintmanager.cpp      1082 0x7ffff79ce932
11  QWidgetRepaintManager::paintAndFlush           qwidgetrepaintmanager.cpp      1014 0x7ffff79d09d4
12  QWidgetRepaintManager::sync                    qwidgetrepaintmanager.cpp      743  0x7ffff79d11d7
13  QWidgetWindow::event                           qwidgetwindow.cpp              347  0x7ffff7a1cb50
14  QApplicationPrivate::notify_helper             qapplication.cpp               3632 0x7ffff79bfa02
15  QCoreApplication::notifyInternal2              qcoreapplication.cpp           1063 0x7ffff6ecf8d1
16  QGuiApplicationPrivate::processExposeEvent     qguiapplication.cpp            3258 0x7ffff72d232f
17  QWindowSystemInterface::sendWindowSystemEvents qwindowsysteminterface.cpp     1169 0x7ffff72a6b7c
18  QXcbUnixEventDispatcher::processEvents         qxcbeventdispatcher.cpp        63   0x7fffe642227f
19  QEventLoop::exec                               qflags.h                       69   0x7ffff6ece3eb
20  main                                           vgl-max-clients-reached-bug.cc 31   0x5555555562bb
... <More>

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x555555f22ce0 vglfaker::VirtualWin
        win 48234519    Window
        xwa @0x7fffffffc630 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1  vglfaker::VirtualWin::VirtualWin                VirtualWin.cpp                 51    0x7ffff7f3b359
2  vglfaker::WindowHash::initVW                    WindowHash.h                   69    0x7ffff7f10c0b
3  glXMakeCurrent                                  faker-glx.cpp                  1592  0x7ffff7f05746
4  QGLXContext::init                               qglxintegration.cpp            415   0x7fffe4f7998b
5  QXcbGlxIntegration::createPlatformOpenGLContext qxcbglxintegration.cpp         194   0x7fffe4f77261
6  QOpenGLContext::create                          qopenglcontext.cpp             612   0x7ffff7314f76
7  QWidgetPrivate::shareContext                    qwidget.cpp                    12006 0x7ffff79ee942
8  QOpenGLWidgetPrivate::initialize                qopenglwidget.cpp              796   0x7ffff7a1f03d
9  QOpenGLWidget::resizeEvent                      qopenglwidget.cpp              1286  0x7ffff7a1f19b
10 QWidget::event                                  qwidget.cpp                    8809  0x7ffff7a00c65
11 QApplicationPrivate::notify_helper              qapplication.cpp               3632  0x7ffff79bfa02
12 QCoreApplication::notifyInternal2               qcoreapplication.cpp           1063  0x7ffff6ecf8d1
13 QWidgetPrivate::sendPendingMoveAndResizeEvents  qwidget.cpp                    7711  0x7ffff79f8d83
14 QWidgetPrivate::show_helper                     qwidget.cpp                    7762  0x7ffff79fd688
15 QWidgetPrivate::setVisible                      qwidget.cpp                    8111  0x7ffff7a0011d
16 main                                            vgl-max-clients-reached-bug.cc 30    0x555555556299

Locals
        this    @0x555555f22ce0 vglfaker::VirtualWin

1  vglfaker::VirtualWin::~VirtualWin                                        VirtualWin.cpp                 90    0x7ffff7f3b921
2  vglfaker::WindowHash::detach                                             WindowHash.h                   117   0x7ffff7ee5aca
3  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::killEntry Hash.h                         137   0x7ffff7ee6515
4  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::remove    Hash.h                         105   0x7ffff7f11d60
5  vglfaker::WindowHash::remove                                             WindowHash.h                   86    0x7ffff7f10cee
6  DeleteWindow                                                             faker-x11.cpp                  251   0x7ffff7f16b2f
7  XDestroyWindow                                                           faker-x11.cpp                  296   0x7ffff7f17b89
8  QXcbGlxIntegration::createPlatformOpenGLContext                          qxcbglxintegration.cpp         194   0x7fffe4f77261
9  QOpenGLContext::create                                                   qopenglcontext.cpp             612   0x7ffff7314f76
10 QWidgetPrivate::shareContext                                             qwidget.cpp                    12006 0x7ffff79ee942
11 QOpenGLWidgetPrivate::initialize                                         qopenglwidget.cpp              796   0x7ffff7a1f03d
12 QOpenGLWidget::resizeEvent                                               qopenglwidget.cpp              1286  0x7ffff7a1f19b
13 QWidget::event                                                           qwidget.cpp                    8809  0x7ffff7a00c65
14 QApplicationPrivate::notify_helper                                       qapplication.cpp               3632  0x7ffff79bfa02
15 QCoreApplication::notifyInternal2                                        qcoreapplication.cpp           1063  0x7ffff6ecf8d1
16 QWidgetPrivate::sendPendingMoveAndResizeEvents                           qwidget.cpp                    7711  0x7ffff79f8d83
17 QWidgetPrivate::show_helper                                              qwidget.cpp                    7762  0x7ffff79fd688
18 QWidgetPrivate::setVisible                                               qwidget.cpp                    8111  0x7ffff7a0011d
19 main                                                                     vgl-max-clients-reached-bug.cc 30    0x555555556299

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x555555f339b0 vglfaker::VirtualWin
        win 48234521    Window
        xwa @0x7fffffffc670 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1  vglfaker::VirtualWin::VirtualWin                VirtualWin.cpp                 51   0x7ffff7f3b359
2  vglfaker::WindowHash::initVW                    WindowHash.h                   69   0x7ffff7f10c0b
3  glXMakeCurrent                                  faker-glx.cpp                  1592 0x7ffff7f05746
4  QGLXContext::init                               qglxintegration.cpp            415  0x7fffe4f7998b
5  QXcbGlxIntegration::createPlatformOpenGLContext qxcbglxintegration.cpp         194  0x7fffe4f77261
6  QOpenGLContext::create                          qopenglcontext.cpp             612  0x7ffff7314f76
7  QOpenGLWidgetPrivate::initialize                qopenglwidget.cpp              813  0x7ffff7a1edd7
8  QOpenGLWidget::resizeEvent                      qopenglwidget.cpp              1286 0x7ffff7a1f19b
9  QWidget::event                                  qwidget.cpp                    8809 0x7ffff7a00c65
10 QApplicationPrivate::notify_helper              qapplication.cpp               3632 0x7ffff79bfa02
11 QCoreApplication::notifyInternal2               qcoreapplication.cpp           1063 0x7ffff6ecf8d1
12 QWidgetPrivate::sendPendingMoveAndResizeEvents  qwidget.cpp                    7711 0x7ffff79f8d83
13 QWidgetPrivate::show_helper                     qwidget.cpp                    7762 0x7ffff79fd688
14 QWidgetPrivate::setVisible                      qwidget.cpp                    8111 0x7ffff7a0011d
15 main                                            vgl-max-clients-reached-bug.cc 30   0x555555556299

Locals
        this    @0x555555f339b0 vglfaker::VirtualWin

1  vglfaker::VirtualWin::~VirtualWin                                        VirtualWin.cpp                 90   0x7ffff7f3b921
2  vglfaker::WindowHash::detach                                             WindowHash.h                   117  0x7ffff7ee5aca
3  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::killEntry Hash.h                         137  0x7ffff7ee6515
4  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::remove    Hash.h                         105  0x7ffff7f11d60
5  vglfaker::WindowHash::remove                                             WindowHash.h                   86   0x7ffff7f10cee
6  DeleteWindow                                                             faker-x11.cpp                  251  0x7ffff7f16b2f
7  XDestroyWindow                                                           faker-x11.cpp                  296  0x7ffff7f17b89
8  QXcbGlxIntegration::createPlatformOpenGLContext                          qxcbglxintegration.cpp         194  0x7fffe4f77261
9  QOpenGLContext::create                                                   qopenglcontext.cpp             612  0x7ffff7314f76
10 QOpenGLWidgetPrivate::initialize                                         qopenglwidget.cpp              813  0x7ffff7a1edd7
11 QOpenGLWidget::resizeEvent                                               qopenglwidget.cpp              1286 0x7ffff7a1f19b
12 QWidget::event                                                           qwidget.cpp                    8809 0x7ffff7a00c65
13 QApplicationPrivate::notify_helper                                       qapplication.cpp               3632 0x7ffff79bfa02
14 QCoreApplication::notifyInternal2                                        qcoreapplication.cpp           1063 0x7ffff6ecf8d1
15 QWidgetPrivate::sendPendingMoveAndResizeEvents                           qwidget.cpp                    7711 0x7ffff79f8d83
16 QWidgetPrivate::show_helper                                              qwidget.cpp                    7762 0x7ffff79fd688
17 QWidgetPrivate::setVisible                                               qwidget.cpp                    8111 0x7ffff7a0011d
18 main                                                                     vgl-max-clients-reached-bug.cc 30   0x555555556299

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x555555f4e2c0 vglfaker::VirtualWin
        win 48234526    Window
        xwa @0x7fffffffc100 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1   vglfaker::VirtualWin::VirtualWin                VirtualWin.cpp             51   0x7ffff7f3b359
2   vglfaker::WindowHash::initVW                    WindowHash.h               69   0x7ffff7f10c0b
3   glXMakeCurrent                                  faker-glx.cpp              1592 0x7ffff7f05746
4   QGLXContext::init                               qglxintegration.cpp        415  0x7fffe4f7998b
5   QXcbGlxIntegration::createPlatformOpenGLContext qxcbglxintegration.cpp     194  0x7fffe4f77261
6   QOpenGLContext::create                          qopenglcontext.cpp         612  0x7ffff7314f76
7   QPlatformBackingStore::composeAndFlush          qplatformbackingstore.cpp  335  0x7ffff7620e57
8   QXcbBackingStore::composeAndFlush               qxcbbackingstore.cpp       900  0x7fffe64189db
9   QXcbBackingStore::composeAndFlush               qxcbbackingstore.cpp       891  0x7fffe64189db
10  QWidgetRepaintManager::flush                    qwidgetrepaintmanager.cpp  1179 0x7ffff79ccc80
11  QWidgetRepaintManager::flush                    qwidgetrepaintmanager.cpp  1082 0x7ffff79ce932
12  QWidgetRepaintManager::paintAndFlush            qwidgetrepaintmanager.cpp  1014 0x7ffff79d09d4
13  QWidgetRepaintManager::sync                     qwidgetrepaintmanager.cpp  743  0x7ffff79d11d7
14  QWidgetWindow::event                            qwidgetwindow.cpp          347  0x7ffff7a1cb50
15  QApplicationPrivate::notify_helper              qapplication.cpp           3632 0x7ffff79bfa02
16  QCoreApplication::notifyInternal2               qcoreapplication.cpp       1063 0x7ffff6ecf8d1
17  QGuiApplicationPrivate::processExposeEvent      qguiapplication.cpp        3258 0x7ffff72d232f
18  QWindowSystemInterface::sendWindowSystemEvents  qwindowsysteminterface.cpp 1169 0x7ffff72a6b7c
19  QXcbUnixEventDispatcher::processEvents          qxcbeventdispatcher.cpp    63   0x7fffe642227f
20  QEventLoop::exec                                qflags.h                   69   0x7ffff6ece3eb
... <More>

Locals
        this    @0x555555f4e2c0 vglfaker::VirtualWin

1   vglfaker::VirtualWin::~VirtualWin                                        VirtualWin.cpp            90   0x7ffff7f3b921
2   vglfaker::WindowHash::detach                                             WindowHash.h              117  0x7ffff7ee5aca
3   vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::killEntry Hash.h                    137  0x7ffff7ee6515
4   vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::remove    Hash.h                    105  0x7ffff7f11d60
5   vglfaker::WindowHash::remove                                             WindowHash.h              86   0x7ffff7f10cee
6   DeleteWindow                                                             faker-x11.cpp             251  0x7ffff7f16b2f
7   XDestroyWindow                                                           faker-x11.cpp             296  0x7ffff7f17b89
8   QXcbGlxIntegration::createPlatformOpenGLContext                          qxcbglxintegration.cpp    194  0x7fffe4f77261
9   QOpenGLContext::create                                                   qopenglcontext.cpp        612  0x7ffff7314f76
10  QPlatformBackingStore::composeAndFlush                                   qplatformbackingstore.cpp 335  0x7ffff7620e57
11  QXcbBackingStore::composeAndFlush                                        qxcbbackingstore.cpp      900  0x7fffe64189db
12  QXcbBackingStore::composeAndFlush                                        qxcbbackingstore.cpp      891  0x7fffe64189db
13  QWidgetRepaintManager::flush                                             qwidgetrepaintmanager.cpp 1179 0x7ffff79ccc80
14  QWidgetRepaintManager::flush                                             qwidgetrepaintmanager.cpp 1082 0x7ffff79ce932
15  QWidgetRepaintManager::paintAndFlush                                     qwidgetrepaintmanager.cpp 1014 0x7ffff79d09d4
16  QWidgetRepaintManager::sync                                              qwidgetrepaintmanager.cpp 743  0x7ffff79d11d7
17  QWidgetWindow::event                                                     qwidgetwindow.cpp         347  0x7ffff7a1cb50
18  QApplicationPrivate::notify_helper                                       qapplication.cpp          3632 0x7ffff79bfa02
19  QCoreApplication::notifyInternal2                                        qcoreapplication.cpp      1063 0x7ffff6ecf8d1
20  QGuiApplicationPrivate::processExposeEvent                               qguiapplication.cpp       3258 0x7ffff72d232f
... <More>

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x555555f4e2c0 vglfaker::VirtualWin
        win 48234517    Window
        xwa @0x7fffffffc1c0 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1   vglfaker::VirtualWin::VirtualWin               VirtualWin.cpp                 51   0x7ffff7f3b359
2   vglfaker::WindowHash::initVW                   WindowHash.h                   69   0x7ffff7f10c0b
3   glXMakeCurrent                                 faker-glx.cpp                  1604 0x7ffff7f05802
4   QGLXContext::makeCurrent                       qglxintegration.cpp            571  0x7fffe4f78ec4
5   QOpenGLContext::makeCurrent                    qopenglcontext.cpp             992  0x7ffff73123cc
6   QPlatformBackingStore::composeAndFlush         qscopedpointer.h               116  0x7ffff761ffed
7   QXcbBackingStore::composeAndFlush              qxcbbackingstore.cpp           900  0x7fffe64189db
8   QXcbBackingStore::composeAndFlush              qxcbbackingstore.cpp           891  0x7fffe64189db
9   QWidgetRepaintManager::flush                   qwidgetrepaintmanager.cpp      1179 0x7ffff79ccc80
10  QWidgetRepaintManager::flush                   qwidgetrepaintmanager.cpp      1082 0x7ffff79ce932
11  QWidgetRepaintManager::paintAndFlush           qwidgetrepaintmanager.cpp      1014 0x7ffff79d09d4
12  QWidgetRepaintManager::sync                    qwidgetrepaintmanager.cpp      743  0x7ffff79d11d7
13  QWidgetWindow::event                           qwidgetwindow.cpp              347  0x7ffff7a1cb50
14  QApplicationPrivate::notify_helper             qapplication.cpp               3632 0x7ffff79bfa02
15  QCoreApplication::notifyInternal2              qcoreapplication.cpp           1063 0x7ffff6ecf8d1
16  QGuiApplicationPrivate::processExposeEvent     qguiapplication.cpp            3258 0x7ffff72d232f
17  QWindowSystemInterface::sendWindowSystemEvents qwindowsysteminterface.cpp     1169 0x7ffff72a6b7c
18  QXcbUnixEventDispatcher::processEvents         qxcbeventdispatcher.cpp        63   0x7fffe642227f
19  QEventLoop::exec                               qflags.h                       69   0x7ffff6ece3eb
20  main                                           vgl-max-clients-reached-bug.cc 31   0x5555555562bb
... <More>

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x5555559b8830 vglfaker::VirtualWin
        win 48234530    Window
        xwa @0x7fffffffc630 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1  vglfaker::VirtualWin::VirtualWin                VirtualWin.cpp                 51    0x7ffff7f3b359
2  vglfaker::WindowHash::initVW                    WindowHash.h                   69    0x7ffff7f10c0b
3  glXMakeCurrent                                  faker-glx.cpp                  1592  0x7ffff7f05746
4  QGLXContext::init                               qglxintegration.cpp            415   0x7fffe4f7998b
5  QXcbGlxIntegration::createPlatformOpenGLContext qxcbglxintegration.cpp         194   0x7fffe4f77261
6  QOpenGLContext::create                          qopenglcontext.cpp             612   0x7ffff7314f76
7  QWidgetPrivate::shareContext                    qwidget.cpp                    12006 0x7ffff79ee942
8  QOpenGLWidgetPrivate::initialize                qopenglwidget.cpp              796   0x7ffff7a1f03d
9  QOpenGLWidget::resizeEvent                      qopenglwidget.cpp              1286  0x7ffff7a1f19b
10 QWidget::event                                  qwidget.cpp                    8809  0x7ffff7a00c65
11 QApplicationPrivate::notify_helper              qapplication.cpp               3632  0x7ffff79bfa02
12 QCoreApplication::notifyInternal2               qcoreapplication.cpp           1063  0x7ffff6ecf8d1
13 QWidgetPrivate::sendPendingMoveAndResizeEvents  qwidget.cpp                    7711  0x7ffff79f8d83
14 QWidgetPrivate::show_helper                     qwidget.cpp                    7762  0x7ffff79fd688
15 QWidgetPrivate::setVisible                      qwidget.cpp                    8111  0x7ffff7a0011d
16 main                                            vgl-max-clients-reached-bug.cc 30    0x555555556299

Locals
        this    @0x5555559b8830 vglfaker::VirtualWin

1  vglfaker::VirtualWin::~VirtualWin                                        VirtualWin.cpp                 90    0x7ffff7f3b921
2  vglfaker::WindowHash::detach                                             WindowHash.h                   117   0x7ffff7ee5aca
3  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::killEntry Hash.h                         137   0x7ffff7ee6515
4  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::remove    Hash.h                         105   0x7ffff7f11d60
5  vglfaker::WindowHash::remove                                             WindowHash.h                   86    0x7ffff7f10cee
6  DeleteWindow                                                             faker-x11.cpp                  251   0x7ffff7f16b2f
7  XDestroyWindow                                                           faker-x11.cpp                  296   0x7ffff7f17b89
8  QXcbGlxIntegration::createPlatformOpenGLContext                          qxcbglxintegration.cpp         194   0x7fffe4f77261
9  QOpenGLContext::create                                                   qopenglcontext.cpp             612   0x7ffff7314f76
10 QWidgetPrivate::shareContext                                             qwidget.cpp                    12006 0x7ffff79ee942
11 QOpenGLWidgetPrivate::initialize                                         qopenglwidget.cpp              796   0x7ffff7a1f03d
12 QOpenGLWidget::resizeEvent                                               qopenglwidget.cpp              1286  0x7ffff7a1f19b
13 QWidget::event                                                           qwidget.cpp                    8809  0x7ffff7a00c65
14 QApplicationPrivate::notify_helper                                       qapplication.cpp               3632  0x7ffff79bfa02
15 QCoreApplication::notifyInternal2                                        qcoreapplication.cpp           1063  0x7ffff6ecf8d1
16 QWidgetPrivate::sendPendingMoveAndResizeEvents                           qwidget.cpp                    7711  0x7ffff79f8d83
17 QWidgetPrivate::show_helper                                              qwidget.cpp                    7762  0x7ffff79fd688
18 QWidgetPrivate::setVisible                                               qwidget.cpp                    8111  0x7ffff7a0011d
19 main                                                                     vgl-max-clients-reached-bug.cc 30    0x555555556299

Locals
        dpy_    @0x55555559a0d0 Display
        this    @0x555555f37d40 vglfaker::VirtualWin
        win 48234532    Window
        xwa @0x7fffffffc670 XWindowAttributes
        xwa@1   <optimized out>
        xwa@2   <optimized out>

1  vglfaker::VirtualWin::VirtualWin                VirtualWin.cpp                 51   0x7ffff7f3b359
2  vglfaker::WindowHash::initVW                    WindowHash.h                   69   0x7ffff7f10c0b
3  glXMakeCurrent                                  faker-glx.cpp                  1592 0x7ffff7f05746
4  QGLXContext::init                               qglxintegration.cpp            415  0x7fffe4f7998b
5  QXcbGlxIntegration::createPlatformOpenGLContext qxcbglxintegration.cpp         194  0x7fffe4f77261
6  QOpenGLContext::create                          qopenglcontext.cpp             612  0x7ffff7314f76
7  QOpenGLWidgetPrivate::initialize                qopenglwidget.cpp              813  0x7ffff7a1edd7
8  QOpenGLWidget::resizeEvent                      qopenglwidget.cpp              1286 0x7ffff7a1f19b
9  QWidget::event                                  qwidget.cpp                    8809 0x7ffff7a00c65
10 QApplicationPrivate::notify_helper              qapplication.cpp               3632 0x7ffff79bfa02
11 QCoreApplication::notifyInternal2               qcoreapplication.cpp           1063 0x7ffff6ecf8d1
12 QWidgetPrivate::sendPendingMoveAndResizeEvents  qwidget.cpp                    7711 0x7ffff79f8d83
13 QWidgetPrivate::show_helper                     qwidget.cpp                    7762 0x7ffff79fd688
14 QWidgetPrivate::setVisible                      qwidget.cpp                    8111 0x7ffff7a0011d
15 main                                            vgl-max-clients-reached-bug.cc 30   0x555555556299

Locals
        this    @0x555555f37d40 vglfaker::VirtualWin

1  vglfaker::VirtualWin::~VirtualWin                                        VirtualWin.cpp                 90   0x7ffff7f3b921
2  vglfaker::WindowHash::detach                                             WindowHash.h                   117  0x7ffff7ee5aca
3  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::killEntry Hash.h                         137  0x7ffff7ee6515
4  vglfaker::Hash<char *, unsigned long, vglfaker::VirtualWin *>::remove    Hash.h                         105  0x7ffff7f11d60
5  vglfaker::WindowHash::remove                                             WindowHash.h                   86   0x7ffff7f10cee
6  DeleteWindow                                                             faker-x11.cpp                  251  0x7ffff7f16b2f
7  XDestroyWindow                                                           faker-x11.cpp                  296  0x7ffff7f17b89
8  QXcbGlxIntegration::createPlatformOpenGLContext                          qxcbglxintegration.cpp         194  0x7fffe4f77261
9  QOpenGLContext::create                                                   qopenglcontext.cpp             612  0x7ffff7314f76
10 QOpenGLWidgetPrivate::initialize                                         qopenglwidget.cpp              813  0x7ffff7a1edd7
11 QOpenGLWidget::resizeEvent                                               qopenglwidget.cpp              1286 0x7ffff7a1f19b
12 QWidget::event                                                           qwidget.cpp                    8809 0x7ffff7a00c65
13 QApplicationPrivate::notify_helper                                       qapplication.cpp               3632 0x7ffff79bfa02
14 QCoreApplication::notifyInternal2                                        qcoreapplication.cpp           1063 0x7ffff6ecf8d1
15 QWidgetPrivate::sendPendingMoveAndResizeEvents                           qwidget.cpp                    7711 0x7ffff79f8d83
16 QWidgetPrivate::show_helper                                              qwidget.cpp                    7762 0x7ffff79fd688
17 QWidgetPrivate::setVisible                                               qwidget.cpp                    8111 0x7ffff7a0011d
18 main                                                                     vgl-max-clients-reached-bug.cc 30   0x555555556299

As can be seen there are two instances above where a VirtualWin is constructed without being destructed after. It can also be seen that in these cases we hit a slightly different path in glXMakeCurrent() where we hit call winhash.initVW() in server/faker-glx.cpp:1604, which is this code path:

else if(!glxdhash.getCurrentDisplay(drawable))
{
    // Apparently it isn't a Pbuffer or a Pixmap, so it must be a window
    // that was created in another process.
    winhash.add(dpy, drawable);
    vw = winhash.initVW(dpy, drawable, config);
    if(vw)
    {
        drawable = vw->updateGLXDrawable();
        vw->setDirect(direct);
    }
}

That looks wrong - the example that @JHagemeister posted does not reference any windows created in another process.

[tsondergaard]

I've debugged a bit further and Qt (5.15 at least) uses xcb_create_window() in QXcbWindow::create() in qtbase/src/plugins/platforms/xcb/qxcbwindow.cpp to create the window that later hits the "Apparently it isn't a Pbuffer or a Pixmap, so it must be a window that was created in another process" path in glXMakeCurrent() in virtualgl/server/faker-glx.cpp. The window is destroyed with xcb_destroy_window() in QXcbWindow::destroy() called from ~QXcbWindow().

Neither xcb_create_window() or xcb_destroy_window() are interposed by virtualgl. I think interposing these two might fix the problem.

It also is not ideal that hitting the "Apparently it isn't a Pbuffer or a Pixmap, so it must be a window that was created in another process" path in glXMakeCurrent() in virtualgl/server/faker-glx.cpp leads to a leaked VirtualWin. Is there a way virtualgl could monitor the lifetime of the associated X window?

[dcommander]

Yeah, I am trying to interpose the relevant XCB functions as we speak. Unfortunately, I don't know of any way to reliably track the life of the window without interposing the functions that are used to create and destroy it.

[tsondergaard]

I have created a minimal PoC fix to demonstrate that a minimal fix can be achieved just by interposing xcb_destroy_window:

From a139995586e8332c50b99bcc8a792f61ecd486b5 Mon Sep 17 00:00:00 2001
From: Thomas Sondergaard <thomas.sondergaard@mi.medical.canon>
Date: Wed, 30 Mar 2022 18:21:58 +0200
Subject: [PATCH] Fix leak of VirtualWin's when window is destroyed with xcb
 (#195)

This just a Proof of Concept to demonstrate that the leak is fixed by
interposing xcb_destroy_window(). I expect the real fix will be more
like the faker for XDestroyWindow().
---
 server/faker-mapfile.c | 1 +
 server/faker-sym.h     | 3 +++
 server/faker-xcb.cpp   | 7 +++++++
 3 files changed, 11 insertions(+)

diff --git a/server/faker-mapfile.c b/server/faker-mapfile.c
index 69be35ba..332e3708 100644
--- a/server/faker-mapfile.c
+++ b/server/faker-mapfile.c
@@ -163,6 +163,7 @@

        /* XCB */
        #ifdef FAKEXCB
+        xcb_destroy_window;
        xcb_get_extension_data;
        xcb_glx_query_version;
        xcb_glx_query_version_reply;
diff --git a/server/faker-sym.h b/server/faker-sym.h
index 11f72b8d..a7b07614 100644
--- a/server/faker-sym.h
+++ b/server/faker-sym.h
@@ -717,6 +717,9 @@ SYMDEF(dlopen);

 // XCB functions

+FUNCDEF2(xcb_void_cookie_t, xcb_destroy_window,
+    xcb_connection_t *, conn, xcb_window_t, window, xcb_destroy_window)
+
 FUNCDEF2(const xcb_query_extension_reply_t *, xcb_get_extension_data,
    xcb_connection_t *, conn, xcb_extension_t *, ext, xcb_get_extension_data)

diff --git a/server/faker-xcb.cpp b/server/faker-xcb.cpp
index 781712bd..68a5ffb1 100644
--- a/server/faker-xcb.cpp
+++ b/server/faker-xcb.cpp
@@ -28,6 +28,13 @@

 extern "C" {

+xcb_void_cookie_t
+xcb_destroy_window (xcb_connection_t *conn, xcb_window_t window)
+{
+    winhash.remove(xcbconnhash.getX11Display(conn), window);
+    return _xcb_destroy_window(conn, window);
+}
+

 const xcb_query_extension_reply_t *
    xcb_get_extension_data(xcb_connection_t *conn, xcb_extension_t *ext)
-- 
2.34.1

[dcommander]

I think it will be cleaner to interpose the creation functions as well, so we can provide tracing output. Stand by.

[tsondergaard]

I agree, I wrote the PoC above before I saw your comment, so I just thought I'd post it here. Thanks for the amazingly quick response and what seems like a great library!

[dcommander]

Should be fixed now

[tsondergaard]

Will will give it a thorough testing tomorrow. Thanks again!

[dcommander]

NOTE: SourceForge appears to be down, which is preventing the pre-release build from completing. For now, you'll have to build VGL from source in order to get the fix.

[tsondergaard]

The patched version does indeed fix our problem. Thanks again.