Build Issues on OS X El Capitan

[mikelevins]

Pursuant to this thread on Reddit: https://www.reddit.com/r/lisp/comments/4lo6rl/best_gui_tool_for_common_lisp/

...I'm trying again to install qtools. This time, instead of frobbing around to try to fix it when it fails, I'm just going to report errors I encounter, in hopes that the information will be helpful. Here's how I got the above error:

Working on OSX El Capitan, version 10.11.5:

1. brew remove qt
2. delete the previous (failed) quicklisp installs of qtools and common-qt
3. delete ~/.cache/common-lisp
4. replace the installed SBCL 1.3.0 with a new copy of SBCL 1.3.6 built locally with --fancy (all tests pass)
5. confirm that the installed SBCL is 64-bit
6. read through qt-libs README again
7. confirm that I have Xcode command-line tools installed (from pkgutil: version 7.3.1.0.1.161711523)
8. uninstall the official distribution of Qt 5.6 using Qt Maintenance Tool.app
9. brew install qt (installs qt version qt-4.8.7_2.el_capitan.bottle.tar.gz)
10. in sbcl: (ql:quickload :qt-libs)

debugger invoked on a SIMPLE-ERROR in thread#<THREAD "main thread" RUNNING {1002FCE5E3}>:  SHA3 file mismatch for "/var/folders/tc/yfmtslkj2d558kyld5k649qw0000gn/T/qt4-archive.1.3.zip"!Expected 2X264J3S1U1GZ1R6V352G6O4Y202XA2I481R11205K2K455C3W3V156I5P82S3X1T183NBC5M1FF4XM355T6U6E3I644B6D6944723P6L5G4L3U1P442Z3QTgot 1TAC5V69S6B3K6U5S21182W1Q3O5G625848296A5U4T4Q6U226H3U4323634N6B472L3A675V5G3YO2E4R3G2A66221M3T6Y3E4L1G5O6W6Y6R6F68S6Y5W4L

I'll keep the terminal around in case you want more of the output.

[Shinmera]

The issue you're hitting here is twofold. The first is what it's erroring about, namely that it downloaded an archive from the network, and failed to verify its integrity. This could mean three things: first, that the download really did not complete successfully, second, that the hash qt-libs has saved in its sources is wrong, or third, that it was trying to download the wrong thing.

However, since you already installed Qt locally with brew, it shouldn't try to download an archive for qt4 from the net at all, so there's something else amiss with its autodetection system too. During the loading of qt-libs it should print something like "> Detected Homebrew" if it successfully detects homebrew being installed on your system. Can you find that message? Is your homebrew installed in the default location /usr/local/?

I'll see about reproducing your issue on my local system today.

[mikelevins]

On May 31, 2016, at 2:09 AM, Nicolas Hafner notifications@github.com wrote:

The issue you're hitting here is twofold. The first is what it's erroring about, namely that it downloaded an archive from the network, and failed to verify its integrity. This could mean three things: first, that the download really did not complete successfully, second, that the hash qt-libs has saved in its sources is wrong, or third, that it was trying to download the wrong thing.

However, since you already installed Qt locally with brew, it shouldn't try to download an archive from the net at all, so there's something else amiss with its autodetection system too. During the loading of qt-libs it should print something like "> Detected Homebrew" if it successfully detects homebrew being installed on your system. Can you find that message? Is your homebrew installed in the default location /usr/local/?

I'll see about reproducing your issue on my local system today.

Excellent; thank you. Perhaps we can get it working at last, and then I can go through the same thing with Linux Mint and Windows 10 :-). (The whole pont of the exercise is to write an updated version of Alpaca, but this time one that works on Linux and WIndows as well as OSX.)

Qt is here:

/usr/local/Cellar/qt/4.8.7_2

qmake and friends are linked to /usr/local/bin.

[mikelevins]

On May 31, 2016, at 2:09 AM, Nicolas Hafner notifications@github.com wrote:

During the loading of qt-libs it should print something like "> Detected Homebrew" if it successfully detects homebrew being installed on your system. Can you find that message?

Yes. Here's the whole context:

; Loading "qt-libs" .................................................. [package cl-ppcre]................................ .................................................. [package qt-lib-generator]..

Detected Homebrew. ...................... [package alexandria.0.dev]........................ .................................................. [package babel-encodings]......................... [package babel]................................... .................................................. [package cffi-sys]................................ [package cffi].................................... .................................................. [package cffi-features]........................... [package qt-libs].. ==> downloading #<BUILD-SYSTEM "qt4"> . -> Downloading https://github.com/Shinmera/qt4/archive/qt-libs1.1.3.zip to /var/folders/tc/yfmtslkj2d558kyld5k649qw0000gn/T/qt4-archive.1.3.zip To load "drakma": Load 1 ASDF system: drakma ; Loading "drakma" .............................. [package puri].................................... [package cl-base64]............................... [package impl-specific-gray]...................... [package trivial-gray-streams].................... [package chunga].................................. [package flexi-streams]........................... .................................................. .................................................. .................................................. [package chipz]................................... [package usocket]................................. [package bordeaux-threads]........................ [package trivial-garbage]......................... [package cl+ssl].................................. [package drakma]........... ...To load "sha3": Load 1 ASDF system: sha3 ; Loading "sha3" .................... [package sha3]..........

debugger invoked on a SIMPLE-ERROR in thread

<THREAD "main thread" RUNNING {1002FCE5E3}>:

SHA3 file mismatch for "/var/folders/tc/yfmtslkj2d558kyld5k649qw0000gn/T/qt4-archive.1.3.zip"! Expected 2X264J3S1U1GZ1R6V352G6O4Y202XA2I481R11205K2K455C3W3V156I5P82S3X1T183NBC5M1FF4XM355T6U6E3I644B6D6944723P6L5G4L3U1P442Z3QT got 1TAC5V69S6B3K6U5S21182W1Q3O5G625848296A5U4T4Q6U226H3U4323634N6B472L3A675V5G3YO2E4R3G2A66221M3T6Y3E4L1G5O6W6Y6R6F68S6Y5W4L

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name): 0: [CONTINUE ] I am sure that this is fine. 1: [RETRY ] Retry downloading. 2: [RETRY ] Retry downloading #<BUILD-SYSTEM "qt4">. 3: [ACCEPT ] Continue, treating downloading #<BUILD-SYSTEM "qt4"> as having been successful. 4: Retry completing compilation for #<SYSTEM "qt-libs">. 5: Continue, treating completing compilation for #<SYSTEM "qt-libs"> as having been successful. 6: Retry ASDF operation. 7: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the configuration. 8: [ABORT ] Give up on "qt-libs" 9: Exit debugger, returning to top level.

(QT-LIB-GENERATOR:SAFELY-DOWNLOAD-FILE "https://github.com/Shinmera/qt4/archive/qt-libs1.1.3.zip" #<PATHNAME (with no namestring) :HOST #<SB-IMPL::UNIX-HOST {10003DC3E3}> :DEVICE NIL :DIRECTORY (:ABSOLUTE "var" "folders" "tc" "yfmtslkj2d558kyld5k649qw0000gn" "T") :NAME "qt4-archive" :TYPE "1.3.zip" :VERSION NIL> #(105 78 163 136 66 52 35 63 247 113 88 240 ...)) 0]

[Shinmera]

I've pushed a patch to fix the checksum for the Qt4 archive, and to disable downloading of the Qt4 sources altogether, since there's no point to that anyway. This should at least get you through your first issue.

However, the building of smokeqt too crashes for me due to some weirdness with library search paths. I'll check back once I got that working too.

[mikelevins]

On May 31, 2016, at 3:21 AM, Nicolas Hafner notifications@github.com wrote:

I've pushed a patch to fix the checksum for the Qt4 archive, and to disable downloading of the Qt4 sources altogether, since there's no point to that anyway. This should at least get you through your first issue.

However, the building of smokeqt too crashes for me due to some weirdness with library search paths. I'll check back once I got that working too.

Great!

I'm pretty sure I've seen the smoke problem before too. At least I've seen some smoke problem. It's hard to be sure because I've tried this several times and I've seen a lot of different problems, but since we're actively talking about it now, we ought to be able to clear things up.

Thanks for the help.

[Shinmera]

Ok, so the building is hitting an issue that was introduced with El Capitan. A "security feature" that strips DYLD_LIBRARY_PATH from child-shells, which breaks the build. Might have to require disabling SIP to build it.

[mikelevins]

On May 31, 2016, at 4:13 AM, Nicolas Hafner notifications@github.com wrote:

Ok, so the building is hitting an issue that was introduced with El Capitan. A "security feature" that strips DYLD_LIBRARY_PATH from child-shells, which breaks the build.

Well, at least that's a comprehensible explanation.

[Shinmera]

I can remedy that initial problem by fixing up the path in smokegen with install_name_tool. But, then it hits another issue where it tries to dlopen a libgenerator_smoke.dylib, which does not exist anywhere in the smokegen distribution. The only similar thing is generator_smoke.so. I have no idea what is up with that, but it smells dangerously like it's generating bad code now in addition to everything else going south.

[mikelevins]

On May 31, 2016, at 4:28 AM, Nicolas Hafner notifications@github.com wrote:

I can remedy that initial problem by fixing up the path in smokegen with install_name_tool. But, then it hits another issue where it tries to dlopen a libgenerator_smoke.dylib, which does not exist anywhere in the smokegen distribution. The only similar thing is generator_smoke.so. I have no idea what is up with that, but it smells dangerously like it's generating bad code now in addition to everything else going south.

You have my sincere sympathies. If there's anything I can do to help, let me know.

[Shinmera]

Now that I'm finally home I could go ahead and disable SIP. Promptly enough, things built fine. So, I'm not sure what to do in this situation. There's three ways forward:

1. Figure out how to make this work with SIP active.
2. Require SIP to be disabled in order to build and work with qt-libs.
3. Disable building on OS X by default and instead use prebuilt binaries.

Ranging in my preferences from most awful to least awful. The third choice being "not so bad" in my eyes since that's already what's happening on Windows anyway and things are downloaded over HTTPs and are integrity verified, so it should be "mostly ok" in terms of security (putting aside the fact that the entirety of the QL ecosystem is insecure as of now anyway). The bigger problem with it is that I don't know for certain how compatible it is with ABI/API changes from OS X' side.

[mikelevins]

I have SIP disabled, so that's presumably not what' blocking me. Ofcourse, that doesn't affect your choices.

Your option 3 doesn't seem that bad to me, either. you aren't making it impossible to build on OSX. You're just acknowledging that Apple has made it inconvenient, and softening that blow by providing prebuilt binaries. Is it ideal? No, of course not. But it's not unworkable. If someone really wants to build from source, they can disable SIP (I leave it disabled anyway most of the time for various reasons).

About the ABI and API changes I don't know anything useful. I'm eager to test whatever you come up with,though.

[Shinmera]

Ok. If you could test again, that'd be great. Simply pull the qt-libs repo from here to your local-projects folder and try loading it again after restarting your lisp. Building has succeeded for me fully now, so I need to know what the next roadblock is for you, if any.

[mikelevins]

Retrying now.

[mikelevins]

Okay, the new download of qt-libs seems to have finished loading. I'm going to try cloning and loading qtools now.

[Shinmera]

Alright, keeping my fingers crossed!

[mikelevins]

qtools loaded successfully. groveling around looking for a quick smoke test I can run...

[Shinmera]

You can load the qtools-helloworld system and launch it with (qtools-helloworld:main)

[mikelevins]

asdf is complaining that it can't find qtcore. I'm looking to see how to satisfy that dependency without invoking quicklisp and possibly triggering a reload of the qt stuff in quicklisp...

[Shinmera]

You have to add the smoke folder in the qtools root to your ASDF search path. If you have it in local-projects, just doing a (ql:register-local-projects) would fix that.

[mikelevins]

Here's what I've got now:

• (asdf:load-system :qtools-helloworld)

debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR in thread

<THREAD "main thread" RUNNING {1002FCE873}>:

Unable to load foreign library (LIBQTCORE.DYLIB-733). Error opening shared object "libQtCore.dylib": dlopen(libQtCore.dylib, 10): image not found.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name): 0: [RETRY ] Try loading the foreign library again. 1: [USE-VALUE ] Use another library instead. 2: [RETRY ] Retry completing compilation for #<SMOKE-MODULE-SYSTEM "qtcore">. 3: [ACCEPT ] Continue, treating completing compilation for #<SMOKE-MODULE-SYSTEM "qtcore"> as having been successful. 4: Retry ASDF operation. 5: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the configuration. 6: [ABORT ] Exit debugger, returning to top level.

(CFFI::FL-ERROR "Unable to load foreign library (~A).~% ~A" #:LIBQTCORE.DYLIB-733 "Error opening shared object \"libQtCore.dylib\": dlopen(libQtCore.dylib, 10): image not found.") 0]

[Shinmera]

Is there a libQtCore.dylib in the standalone folder in qt-libs?

[mikelevins]

qt-libs contains:

/ .travis.yml archives.lisp package.lisp qt-libs.lisp smokeqt.lisp ../ LICENSE commonqt.lisp pathnames.lisp qt4.lisp system.lisp .git/ README.md dylib.lisp qt-lib-generator.asd setenv.lisp toolkit.lisp .gitignore about.html osx.lisp qt-libs.asd smokegen.lisp

[mikelevins]

I'm searching for where qt-libs put the downloads....

[Shinmera]

During the setup process it should have copied the libraries over to a directory that is referenced in qt-libs:*standalone-libs-dir*

[mikelevins]

That seems to be /Users/mikel/quicklisp/dists/quicklisp/software/qt-libs-20160318-git/standalone/

[mikelevins]

...and sure enough, there's a standalone directory there with a mess of dylibs

[mikelevins]

Interestingly, libQtCore.dylib is not one of them

[mikelevins]

Here's the answer to the next logical question you might be likely to ask:

libcommonqt.dylib libsmokebase.dylib libsmokephonon.dylib libsmokeqtcore.dylib libsmokeqtdeclarative.dylib libsmokeqtgui.dylib libsmokeqthelp.dylib libsmokeqtmultimedia.dylib libsmokeqtnetwork.dylib libsmokeqtopengl.dylib libsmokeqtscript.dylib libsmokeqtsql.dylib libsmokeqtsvg.dylib libsmokeqttest.dylib libsmokeqtuitools.dylib libsmokeqtwebkit.dylib libsmokeqtxml.dylib libsmokeqtxmlpatterns.dylib

[mikelevins]

That's what's in the standalone directory

[Shinmera]

The path is telling me that it is not loading qt-libs from the local-projects folder, but instead is still picking the quicklisp source. This would also explain why it didn't copy the Qt libraries over quite right, since the path for those was not adapted for 4.8.7_2 yet.

[mikelevins]

I can nuke the quicklisp install and try again. I'll try to avoid invoking quicklisp for anything -- I had to use it to get a couple of things that were missing, e.g. form-fiddle, but those should be in place now.

[Shinmera]

Generally I go the other direction where I let QL manage everything, but sure, as long as you can make sure that the qt-libs from git is being loaded.

[mikelevins]

that's why I'm avoiding quicklisp for the moment. obviously I want to go back to using it as soon as possible.

[Shinmera]

Hunh. I've never had problems with load preferences as long as I made sure to keep things in local-projects and use (ql:register-local-projects) when I add new systems to it.

[mikelevins]

Yeah well, I haven't done either of those things with this stuff yet--I was counting on quicklisp to download them for me and only starting messing around with the newer git clone less than an hour ago :-)

[mikelevins]

Oho; now the qt-libs load is creating a standalone directory in the git clone instead of the (now nonexistent) quicklisp copy.

[mikelevins]

...and it contains the library that it couldn't find before.

[Shinmera]

I've had another unfortunate problem on my system where trying to run any kind of Qt function after loading the libraries resulted in a memory fault. I can't say I know where that would stem from or how to go about fixing it besides using the precompiled versions that I prepared on an earlier version of OS X. Those seem to work without a hitch.

So, if you run into that... well, precompiled sources it is for the moment.

[mikelevins]

I got a new error loading helloworld:

; compiling file "/Volumes/pluto/Users/mikel/Shed/qtools/examples/helloworld/helloworld.lisp" (written 31 MAY 2016 07:24:18 AM): ; compiling (DEFPACKAGE #:QTOOLS-HELLOWORLD ...) ; compiling (IN-PACKAGE #:ORG.SHIRAKUMO.QTOOLS.HELLOWORLD) ; compiling (IN-READTABLE :QTOOLS) ; compiling (DEFINE-WIDGET MAIN ...)CORRUPTION WARNING in SBCL pid 3416(tid 140735179964416): Memory fault at 0x224 (pc=0x7fff8de2e547, sp=0x51fe000) The integrity of this image is possibly compromised. Continuing with fingers crossed.

; file: /Volumes/pluto/Users/mikel/Shed/qtools/examples/helloworld/helloworld.lisp ; in: DEFINE-WIDGET MAIN ; (QTOOLS:DEFINE-WIDGET QTOOLS-HELLOWORLD:MAIN (QTOOLS-HELLOWORLD::QWIDGET) NIL) ; ; caught ERROR: ; (during macroexpansion of (DEFINE-WIDGET MAIN ...)) ; Unhandled memory fault at #x224.

; compiling (DEFINE-SUBWIDGET (MAIN BUTTON) ...) debugger invoked on a SIMPLE-ERROR in thread

<THREAD "main thread" RUNNING {1002FCE7C3}>:

There is no class named QTOOLS-HELLOWORLD:MAIN.

[Shinmera]

Yep, that's the memory fault I was just talking about alright. I'll see what I can find out about that.

[mikelevins]

At least it's new!

I'm going to have to go to sleep soon. I'm running on fumes.

[Shinmera]

Alright, have a good night!

[mikelevins]

Thanks very much for all the help. I'm feeling optimistic about qtools.

[Shinmera]

I've tried compiling against Ports instead of Brew as well and it still memfaults. At this point I can't explain what the problem is beyond OS X being a newer version and that causing a problem. However, since the old libraries seem to work, you can continue on with those for now. In order to get them, proceed as follows:

• Delete the qt-libs ASDF cache folder
• Delete the standalone folder in the qt-libs directory
• Pull qt-libs to the newest version from git
• Load qt-libs again

It should now automatically grab the precompiled libraries.

[mikelevins]

ah, gotcha

[mikelevins]

qtools-helloworld worked. At last! Thank you.

Now I will tinker with qtools and qtools-ui and see i I can sketch out an updated version of Alpaca, and I'll see if I can get the same process to work on Linux.

Thanks very much for your help.

[mikelevins]

I tried three of the qtools examples on SBCL on Linux Mint, and they worked perfectly.

Thanks again.

[mikelevins]

Okay, I've now tried helloworld and opengl on CCL on both OSX El Capitan and Linux Mint Rosa. They work fine. I seem to be in business.

[Shinmera]

Fantastic! Let me know if you come across anything else.

[mikelevins]

On Jun 1, 2016, at 12:25 AM, Nicolas Hafner notifications@github.com wrote:

Fantastic! Let me know if you come across anything else.

Will do. Thanks one more time. Feeling very liberated right now. :-)