Built binary attempts to load UIOP from hard disk

[phoe]

Windows x64, SBCL 1.4.14

ASDF options for the system being built:

  :defsystem-depends-on (:deploy)
  :build-operation "deploy-op"
  :build-pathname "ect-gui"
  :entry-point "ect-gui:main"
  :depends-on (:qtools :qtcore :qtgui :alexandria :cl-json :asdf :uiop
               :drakma :local-time :cl-ppcre :trivial-backtrace)

I compiled as user bar and sent the binary to user foo. When they run the machine, they get the following error:

 ==> Performing warm boot.
   -> Runtime directory is C:/Users/foo/Desktop/bin/
   -> Resource directory is C:/Users/foo/Desktop/bin/
 ==> Running boot hooks.
   -> Loading smoke module QTCORE.
WARNING:
   ASDF 3.3.1 (from NIL), UIOP NIL (from C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd)
   -> Loading smoke module QTGUI.
 ==> Running Qtools boot hooks.
 ==> Reloading foreign libraries.
   -> Loading foreign library #<LIBRARY LIBEAY32>.
   -> Loading foreign library #<LIBRARY LIBSSL>.
 ==> Launching application.
 ==> Encountered unhandled error: Error while trying to load definition for system uiop from pathname C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd: Couldn't load #P"C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd": file does not exist.
;
; compilation unit aborted
;   caught 1 fatal ERROR condition
 ==> Running quit hooks. 

[phoe]

From the first glance, it seems like UIOP is not bundled in the deployed Lisp image for whatever reason, so it attempts to load it from hard drive. And, at that, it fails, because the path is no longer valid.

[Shinmera]

Why would it try to load it anyway though. Do you invoke ASDF somewhere?

And it can't not bundle it. It does an image dump with ASDF. UIOP is already in the image at that point.

[phoe]

Nope. I don't invoke ASDF anywhere in my user code.

And, if UIOP is already in the image (which is obvious and logical), why does it attempt to load it anyway and why does it say ASDF 3.3.1 (from NIL), UIOP NIL (from C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd) ?

[Shinmera]

Fucked if I know, I didn't write the clusterfuck that is ASDF.

[Shinmera]

First thing to do would be to get a backtrace to see where ASDF is invoked.

[phoe]

How do I tell Deploy to automatically print a backtrace on unhandled error?

[Shinmera]

You don't. You make it enter the debugger and print it yourself.

[phoe]

Backtrace:

debugger invoked on a LOAD-SYSTEM-DEFINITION-ERROR in thread #<THREAD "main thread" RUNNING {1002260613}>: Error while trying to load definition for system uiop from pathname C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd: Couldn't load #P"C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd": file does not exist.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry #<DEFINE-OP > on #<PACKAGE-SYSTEM "uiop">.
  1: [ACCEPT                       ] Continue, treating #<DEFINE-OP > on #<PACKAGE-SYSTEM "uiop"> as having been successful.
  2:                                 Retry ASDF operation.
  3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the configuration.
  4:                                 Retry ASDF operation.
  5:                                 Retry ASDF operation after resetting the configuration.
  6: [EXIT                         ] Exit.
  7: [ABORT                        ] Exit from the current thread.

((FLET "H0" :IN PERFORM) #<SB-INT:SIMPLE-FILE-ERROR "~@<Couldn't load ~S: file does not exist.~@:>" {1009BF6743}>)
   error finding frame source: Bogus form-number: the source file has probably changed too much to cope with.
   source: NIL
0] BACKTRACE

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1002260613}>
0: ((FLET "H0" :IN PERFORM) #<SB-INT:SIMPLE-FILE-ERROR "~@<Couldn't load ~S: file does not exist.~@:>" {1009BF6743}>)
1: (SB-KERNEL::%SIGNAL #<SB-INT:SIMPLE-FILE-ERROR "~@<Couldn't load ~S: file does not exist.~@:>" {1009BF6743}>)
2: (ERROR SB-INT:SIMPLE-FILE-ERROR :PATHNAME #P"C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd" :FORMAT-CONTROL "~@<Couldn't load ~S: file does not exist.~@:>" :FORMAT-ARGUMENTS (#P"C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd"))
3: (LOAD #P"C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd" :VERBOSE NIL :PRINT NIL :IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :UTF-8)
4: (CALL-WITH-MUFFLED-CONDITIONS #<CLOSURE (LAMBDA NIL :IN LOAD*) {1009BF60BB}> ("Overwriting already existing readtable ~S." #(#:FINALIZERS-OFF-WARNING :ASDF-FINALIZERS)))
5: ((FLET "THUNK" :IN PERFORM))
6: (SB-IMPL::%WITH-STANDARD-IO-SYNTAX #<CLOSURE (FLET "THUNK" :IN PERFORM) {32E37B}>)
7: ((:METHOD PERFORM (DEFINE-OP SYSTEM)) #<DEFINE-OP > #<PACKAGE-SYSTEM "uiop">) [fast-method]
8: ((SB-PCL::EMF PERFORM) #<unused argument> #<unused argument> #<DEFINE-OP > #<PACKAGE-SYSTEM "uiop">)
9: ((LAMBDA NIL :IN ASDF/ACTION:CALL-WHILE-VISITING-ACTION))
10: ((:METHOD PERFORM-WITH-RESTARTS :AROUND (T T)) #<DEFINE-OP > #<PACKAGE-SYSTEM "uiop">) [fast-method]
11: ((:METHOD PERFORM-PLAN (T)) #<SEQUENTIAL-PLAN {1008D6D403}>) [fast-method]
12: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT))
13: ((:METHOD PERFORM-PLAN :AROUND (T)) #<SEQUENTIAL-PLAN {1008D6D403}>) [fast-method]
14: ((:METHOD OPERATE (OPERATION COMPONENT)) #<LOAD-OP > #<QT-LIBS:FOREIGN-LIBRARY-SYSTEM "qtcore"> :PLAN-CLASS NIL :PLAN-OPTIONS NIL) [fast-method]
15: ((SB-PCL::EMF OPERATE) #<unused argument> #<unused argument> #<LOAD-OP > #<QT-LIBS:FOREIGN-LIBRARY-SYSTEM "qtcore"> :VERBOSE NIL)
16: ((LAMBDA NIL :IN OPERATE))
17: ((:METHOD OPERATE :AROUND (T T)) #<LOAD-OP > #<QT-LIBS:FOREIGN-LIBRARY-SYSTEM "qtcore"> :VERBOSE NIL) [fast-method]
18: ((SB-PCL::EMF OPERATE) #<unused argument> #<unused argument> LOAD-OP "qtcore" :VERBOSE NIL)
19: ((LAMBDA NIL :IN OPERATE))
20: ((:METHOD OPERATE :AROUND (T T)) LOAD-OP "qtcore" :VERBOSE NIL) [fast-method]
21: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<CLOSURE (LAMBDA NIL :IN OPERATE) {1008D6097B}> :OVERRIDE T :KEY NIL :OVERRIDE-CACHE T :OVERRIDE-FORCING NIL)
22: ((LAMBDA NIL :IN OPERATE))
23: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<CLOSURE (LAMBDA NIL :IN OPERATE) {1008D133EB}> :OVERRIDE NIL :KEY NIL :OVERRIDE-CACHE NIL :OVERRIDE-FORCING NIL)
24: ((:METHOD OPERATE :AROUND (T T)) LOAD-OP "qtcore" :VERBOSE NIL) [fast-method]
25: (LOAD-SYSTEM "qtcore" :VERBOSE NIL)
26: (QT:ENSURE-SMOKE :QTCORE)
27: (QTOOLS:ENSURE-QAPPLICATION :NAME NIL :ARGS NIL :MAIN-THREAD NIL)
28: ((LABELS QTOOLS::MAIN :IN QTOOLS::MAIN-WINDOW-EXEC))
29: (QTOOLS::MAIN-WINDOW-EXEC #<FUNCTION (LAMBDA NIL :IN ECT-GUI:MAIN) {10000D528B}> :NAME NIL :QAPPLICATION-ARGS NIL :BLOCKING T :MAIN-THREAD NIL :ON-ERROR #<FUNCTION INVOKE-DEBUGGER> :SHOW T :FINALIZE T :BEFORE-EXEC #<FUNCTION (LAMBDA (ECT-GUI::WINDOW) :IN ECT-GUI:MAIN) {10000D4F5B}> :AFTER-EXEC NIL)
30: ((FLET "CLEANUP-FUN-0" :IN ECT-GUI:MAIN)) [cleanup]
31: (ECT-GUI:MAIN)
32: (DEPLOY::CALL-ENTRY-PREPARED #<FUNCTION ECT-GUI:MAIN> #<SYSTEM "ect-gui"> #<DEPLOY-OP >)
33: ((LAMBDA NIL :IN RESTORE-IMAGE))
34: (CALL-WITH-FATAL-CONDITION-HANDLER #<CLOSURE (LAMBDA NIL :IN RESTORE-IMAGE) {100478E91B}>)
35: ((FLET SB-UNIX::BODY :IN SB-EXT:SAVE-LISP-AND-DIE))
36: ((FLET "WITHOUT-INTERRUPTS-BODY-14" :IN SB-EXT:SAVE-LISP-AND-DIE))
37: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))
38: ("foreign function: #x43136B")
39: ("foreign function: #x403758")

(LOAD-SYSTEM "qtcore" :VERBOSE NIL) seems to invoke an ASDF load for UIOP.

[Shinmera]

So the problem (that I'm responsible for) is that ensure-smoke calls the asdf operation again even though the library is already loaded per the bootup procedure.

[phoe]

If I select the CONTINUE restart, it then tries to load Alexandria, trivial-features, ...

So basically, ASDF gets invoked once again even though it has nowhere to load the files from.

[phoe]

How does this process even work? QT:ENSURE-SMOKE does not seem to call any ASDF operations or load any ASDF systems. https://github.com/commonqt/commonqt/blob/e3eae79b99aaef07f9ccbaae9c9ac32d2a5710e0/info.lisp#L925

[Shinmera]

It works via the *load-library-function* which delegates to qt-libs, which in turn invokes ASDF.

[phoe]

Quick hack: stubbing out ENSURE-LIB-LOADED made it work for me.

(in-package :qt-libs) 
(defun ensure-lib-loaded (file) (declare (ignore file)))

[Shinmera]

Whoops, hit the wrong button. Anyway, the proper fix is to not make ASDF throw a temper tantrum if it gets the sources pulled from under its feet. Maybe using something like the preloaded or immutable systems stuff in ASDF would at least make it avoid reloading for the Qt libraries, though I don't exactly know how these features work.

[Shinmera]

Can you try again with latest qt-libs and qtools? I added some ASDF mangling to prevent it from trying to load stuff.

[phoe]

Will try next week - I do not have my work computer with me at the moment.

[Shinmera]

If this works I can add a more general tool to deploy that allows freezing all known asdf systems in place to avoid this problem in general. I can't make it default however as someone might expect to reload asdf systems on a target machine.

[phoe]

I do not have this environment any longer and therefore cannot reproduce this issue. Should I close it?

[phoe]

Actually hold on a second, I can reproduce this - I just reproduced this on a macOS VM.

I cannot go up to 2.0.2 because of https://github.com/Shinmera/qt-libs/issues/17

[phoe]

I have worked around this by checking out https://github.com/Shinmera/qt-libs/commit/1b20ca0 and cherry-picking https://github.com/Shinmera/qt-libs/commit/6e70e7f on top of that.