Qt4 Plugins Not Loading (Windows 7 x86_x64)

[Shinmera]

From @resttime on January 29, 2016 11:55

Found some (possible) strange behavior: it doesn't seem like qtools/qt-libs can find the plugins in the standalone folder. I did delete the standalone folder and the entire common-lisp cache before the tests. Right now I'm wondering if it's just a problem on my end (which might be most likely), but after spending a long time investigating the problem, I'm not so sure anymore.

Anyways, if it's just an issue on my end, sorry to bother and feel free to close the issue either way.

System info:

CCL 1.11 x86 and SBCL 1.3.1 x86 Qt 4.8.7 x86 Windows 7 x86_64

Here's some quick test code, if the plugins are loaded then there should be more supported formats:

#| QTools Used

Supported Movie Formats: NIL
Supported Image Formats: (bmp pbm pgm png ppm xbm xpm)

|#
(ql:quickload '(:qtools :qtgui :qtcore))

(defpackage #:test
  (:use :cl+qt)
  (:export :foo))
(in-package #:test)
(named-readtables:in-readtable :qtools)

(define-widget foo (QWidget) ())

;; This also breaks with CCL x86 1.11 if 
;; WITH-MAIN-WINDOW has the key arguement :MAIN-THREAD T
(defun foo ()
  (with-main-window (window (make-instance 'foo))
    (format t "Supported Movie Formats: ~a~%"
        (loop for a in (q+:qmovie-supported-formats)
           collect (q+:data a)))
    (format t "Supported Image Formats: ~a~%"
        (loop for a in (q+:qimagereader-supported-image-formats)
           collect (q+:data a)))))

#| CommonQt Only

Supported Movie Formats: (gif mng)
Supported Image Formats: (bmp gif ico jpeg jpg mng pbm pgm png ppm svg svgz tga
                          tif tiff xbm xpm)
|#
(ql:quickload :qt)

(defpackage #:test
  (:use :cl :qt)
  (:export :bar))
(in-package #:test)

(named-readtables:in-readtable :qt)

(defclass bar ()
  ()
  (:metaclass qt:qt-class)
  (:qt-superclass "QWidget"))

(defmethod initialize-instance :after ((this bar) &key)
  (qt:new this))

(defun bar ()
  (qt:with-main-window (bar (make-instance 'bar))
    (format t "Supported Movie Formats: ~a~%"
        (loop for a in (#_QMovie::supportedFormats)
           collect (#_data a)))
    (format t "Supported Image Formats: ~a~%"
        (loop for a in (#_QImageReader::supportedImageFormats)
           collect (#_data a)))
    (#_show bar)
    (#_hide bar)
    (#_show bar)))

_Copied from original issue: Shinmera/qtools#11_

[Shinmera]

Where does CommonQt grab the libraries for on your system?

[Shinmera]

I remember fixing a bug in relation to this-- qt-libs does properly set the plugin path within Qt, so it should be able to discover them. I'll have to look at this closer, maybe there's some name mangling going on that confuses Qt.

[Shinmera]

From @resttime on January 29, 2016 22:35

Qt 4.8.7: C:\Qt\4.8.7\bin commonqt-libs-20131109: C:\Qt\commonqt-libs

Both locations have been put into my PATH environment variable.

Here are the contents of the standalone folder: http://pastebin.com/DF9dDeaZ

Also, I believe I might have found a bug I'm trying to investigate with CL+SSL which subsequently affects qt-libs on Windows with OpenSSL. If this issue is also valid (rather than an issue on my end), wrapping the contents of DOWNLOAD-FILE of archives.lisp in qt-libs to ignore the error might work, or reimplementing it to not use a stream (:WANT-STREAM NIL) by downloading to a byte vector instead and writing that to a file. Neither seems to be good practice, but I figure most computers can spare the memory on the size of the vector for the latter. I could write this for you if you want, since it seems simple enough that I won't be able to screw it up (again assuming it is a valid issue, not just on my end).

When I comment out (CLOSE INPUT) everything works normally (I remembered to delete cache and standalone folders), and there shouldn't be any corruption since SAFELY-DOWNLOAD-FILE runs a checksum on the files. Overall, probably separate from this issue, but I'm mentioning since there might be a chance of affecting this.

Test Code Separate(?) Issue:

;; Closing the SSL stream breaks, with an undefined alien error
;; so DOWNLOAD-FILE of archives.lisp in qt-libs 
;; also is affected unless the stream never closes.

(ql:quickload "drakma")

(defun test (&optional (test-url "https://google.com"))
  (multiple-value-bind (input status) (drakma:http-request test-url
                               :want-stream T)
    (princ status)
    ;; Leave stream open instead of closing and no problems occur.
    (close input)))

[Shinmera]

The SSL issue is most definitely unrelated to this, since I've tested it on multiple machines myself and haven't heard anyone else having problems with it up to this point either.

Unfortunately I won't have time to take a look at this until next week, but it's definitely on the top of my todo.

[Shinmera]

From @resttime on January 30, 2016 11:12

Alright, glad to hear.

As for the CL+SSL issue on Windows: Found bug, have simple fix, and will notify them. Basically, the CL+SSL library introduced the POSIX close(int) to close the SSL socket stream a few months ago. Simple fix done by replacing that with the Windows closesocket(int) via the #+ macro. Phew.

[Shinmera]

From @resttime on February 1, 2016 7:21

Neat, I solved the mystery while snooping inside the contents of qt-libs.lisp and looking for clues at the Qt4 documentation:

"Qt will not find plugins if they are not stored in the right directory." Source: https://doc.qt.io/qt-4.8/plugins-howto.html

Qt 4.8.7 Plugins: C:\Qt\4.8.7\plugins Directories inside that: http://pastebin.com/p3PFe93Z

Solution is to preserve the folder structure of the plugins when copying them to the standalone folder, and the proper fix for qt-libs would be to replicate the same behaviour. In other words, folders in plugins folder should exist in the standalone folder with their respective plugins inside.

[Shinmera]

This should not be necessary because qt-libs adapts the paths where qt looks for plugins and pushes the standalone directory itself onto that list of paths.

[Shinmera]

From @resttime on February 1, 2016 7:41

(defun set-qt-plugin-paths (&rest paths)
  (funcall (csymb '(qt interpret-call)) "QCoreApplication" "setLibraryPaths"
       (mapcar #'uiop:native-namestring paths)))

Are you referring to SET-QT-PLUGIN-PATHS?

I read a bit more and it seems that the original path which qt-libs changes is the plugins directory in the first place, so I guess because of that if the folder structure doesn't match then it won't work.

"The default path list consists of a single entry, the installation directory for plugins. The default installation directory for plugins is INSTALL/plugins, where INSTALL is the directory where Qt was installed."

Source: https://doc.qt.io/qt-4.8/qcoreapplication.html#addLibraryPath

Also, rather than pushing, looks like setLibraryPaths actually clobbers the original paths.

"Sets the list of directories to search when loading libraries to paths. All existing paths will be deleted and the path list will consist of the paths given in paths."

Source: https://doc.qt.io/qt-4.8/qcoreapplication.html#setLibraryPaths

[Shinmera]

If the libraries do not get explicitly set, then the standard path is:

CL-USER> (qt:interpret-call "QCoreApplication" "libraryPaths")
("/usr/lib/qt4/plugins" "/usr/local/bin")

So, setting the path to the standalone directory should be correct:

CL-USER> (qt-libs:set-qt-plugin-paths qt-libs:*standalone-libs-dir*)
NIL
CL-USER> (qt:interpret-call "QCoreApplication" "libraryPaths")
("/media/PRO/Projects/CL/qt-libs/standalone/")

It's intentional that the original path gets purged, as it could otherwise lead to complications on deployed systems.

The only idea I have right now is that the trailing slash might trip Qt up for some retarded reason. Might want to try adapting the set-qt-plugin-paths function to trim that away.

[Shinmera]

From @resttime on February 1, 2016 9:6

Tested and that wasn't it. I read more on the Qt4 documentation surrounding plugins. Sub-directories are the only thing I'm seeing. Unless you know something else, this is the only conclusion I can reach. Maybe there was a misunderstanding? To restate, the solution is to copy the plugin folders into the inside of the standalone folder containing all the other library files.

plugins/accessible/ -> standalone/accessible/ plugins/bearer/ -> standalone/bearer/ plugins/codecs/ -> standalone/codecs/ etc.

When the application is run, Qt will first treat the application's executable directory as the pluginsbase. For example if the application is in C:\Program Files\MyApp and has a style plugin, Qt will look in C:\Program Files\MyApp\styles

Source: https://doc.qt.io/qt-4.8/deployment-plugins.html

There are several plugin base classes. Derived plugins are stored by default in sub-directories of the standard plugin directory. Qt will not find plugins if they are not stored in the right directory.

Source: https://doc.qt.io/qt-4.8/plugins-howto.html#static-plugins (I should have copied the full paragraph for this)

During development, the directory for plugins is QTDIR/plugins (where QTDIR is the directory where Qt is installed), with each type of plugin in a subdirectory for that type, e.g. styles. If you want your applications to use plugins and you don't want to use the standard plugins path, have your installation process determine the path you want to use for the plugins, and save the path, e.g. using QSettings, for the application to read when it runs. The application can then call QCoreApplication::addLibraryPath() with this path and your plugins will be available to the application. Note that the final part of the path (e.g., styles) cannot be changed.

If you want the plugin to be loadable then one approach is to create a subdirectory under the application and place the plugin in that directory. If you distribute any of the plugins that come with Qt (the ones located in the plugins directory), you must copy the sub-directory under plugins where the plugin is located to your applications root folder (i.e., do not include the plugins directory).

Source: https://doc.qt.io/qt-4.8/plugins-howto.html#locating-plugins

;;; Testing widget
(define-subwidget (main-window loading-animation) (q+:make-qmovie "loading.gif")
  (format t "Loading Animation: ~a~%" (q+:is-valid loading-animation))
  (format t "Supported Formats: ~a~%" (loop for a in (q+:qmovie-supported-formats) collect (q+:data a)))
  (format t "Supported Formats: ~a~%" (loop for a in (q+:qimagereader-supported-image-formats) collect (q+:data a)))
  (format t "Library Path: ~a~%" (q+:qcoreapplication-library-paths))
  (q+:set-movie image-label loading-animation)
  (q+:start loading-animation))

;;; Before Trim
CL-USER> (qtimginu:start)

Loading Animation: NIL
Supported Formats: NIL
Supported Formats: (bmp pbm pgm png ppm xbm xpm)
Library Path: (C:\Users\rest\HOME\quicklisp\dists\quicklisp\software\qt-libs-20151218-git\standalone\)

;;; After Trim

;; qt-libs.lisp
(defun set-qt-plugin-paths (&rest paths)
  (funcall (csymb '(qt interpret-call)) "QCoreApplication" "setLibraryPaths"
       (mapcar #'(lambda (path)
               (string-trim "\\" (uiop:native-namestring path))) paths)))

CL-USER> (qtimginu:start)

Loading Animation: NIL
Supported Formats: NIL
Supported Formats: (bmp pbm pgm png ppm xbm xpm)
Library Path: (C:\Users\rest\HOME\quicklisp\dists\quicklisp\software\qt-libs-20151218-git\standalone)

;;; Copying Plugin Folders into the Standalone Folder
CL-USER> (qtimginu:start)

Loading Animation: T
Supported Formats: (gif mng)
Supported Formats: (bmp gif ico jpeg jpg mng pbm pgm png ppm svg svgz tga tif
                    tiff xbm xpm)
Library Path: (C:\Users\rest\HOME\quicklisp\dists\quicklisp\software\qt-libs-20151218-git\standalone)

[Shinmera]

Aw hell. And here I hoped I could avoid bloody subdirectories. Thanks for the research, I'll see how I can remedy this.

[Shinmera]

I changed things around to copy the plugin directory structure verbatim into the standalone directory. Could you check if it's working correctly now @resttime?

[resttime]

Yup, nice job. I think qt-libs works correctly. However, not with Qtools yet because ENSURE-QAPPLICATION of toolkit.lisp in Qtools uses QT-LIBS:SET-QT-PLUGIN-PATHS but doesn't add the new plugins directory (it only adds the standalone directory).

There are probably many ways to fix it. Here's one: export a new parameter in qt-libs.lisp right under the definition of standalone-libs-dir for usage in other libraries

;;; Export me in qt-libs.lisp in the qt-libs package to use 
;;; with ENSURE-QAPPLICATION in toolkit.lisp of Qtools!
(defvar *standalone-plugins-dir* (relative-dir *standalone-libs-dir* "plugins"))

That might also mean changes to MAKE-QAPPLICATION of qt-libs.lisp and other places too, because I think something in deploy.lisp of Qtools wants to know where the plugins are.

The other solution would be to copy the plugin subdirectories verbatim into the standalone directory instead because the standalone directory is considered to be the plugins directory. Though I assume you do not want both directories to be the same, hence the plugin directory being copied verbatim in the standalone directory?

[Shinmera]

I just added a fix-qt-plugin-paths function. The deployment functions don't need to know about plugins because they simply change the *standalone-libs-dir*. Too bad the quicklisp release went through yesterday, oh well. Either way, since this seems to work now, I'll close it.