SBCL Windows port

[kaveh808]

Port the system to sbcl/Windows/OpenGL.

[JMC-design]

Something to keep in mind for the codebase is that because of windows line endings you cannot use ~ in a format string to skip whitespace till the next character.
Right now the only instance of this is in printing the help which shouldn't be much of an issue since that should probably all be moved to a keybinding store and printed from that. It might be an issue later with docstrings. It would probably be best to deal with this in the codebase and not expect everybody to fool with their git checkouts.

[mikelevins]

Here's the latest kons-9 running on sbcl 2.1.10 on my Surface Pro 8 (Windows 11 Pro, version 21H2). I walked through the demo-kernel.lisp file up to the point you see in the screenshot. Everything seemed to work fine. After taking this screenshot I closed the glfw window, then reopened it and held down the spacebar, and the rectangular prisms containued to rotate as expected.

I would note that the restored window exhibits the same shading or lighting artifact we're seeing on macOS and Linux, so presumably Kaveh's right: the issue is probably in some graphics state that the code is failing to preserve across window inits.

I'll add a document to the repo that explains how I got it working; it was very little effort. It mostly just worked.

[mikelevins]

It’s now pushed to the repo in a branch named "SBCL-win64-tests". It’s just two files with very small changes, but you do need glfw installed. I’ll put the details in a doc.

[mikelevins]

Okay, I added notes on running kons-9 on win64 to the sbcl-win64-tests branch.

[kaveh808]

Excellent, thanks. Send me a PR when you feel it's ready.

[mikelevins]

In case anyone wonders: I’m thinking about the best way to tell SBCL where to find the glfw DLL without essentially hard coding a drive letter and path (suggestions welcome). I think once that’s solved we can merge this tiny branch.

[awolven]

If glfw is installed the from a normal glfw installation you can just do this:

(cffi:define-foreign-library glfw3 (:darwin "libglfw.3.4.dylib") (:windows "glfw3.dll") (:linux "libglfw.so.3"))

for other libraries do this:

(defmacro my-define-foreign-library (name-and-options &body pairs) `(cffi:define-foreign-library ,name-and-options ,@(mapcar (lambda (pair) (list (first pair) (eval (second pair)))) pairs)))

(my-define-foreign-library cimgui (:darwin (asdf/system:system-relative-pathname :vktk "ifc/lib/cimgui.dylib")) (:windows (asdf/system:system-relative-pathname :vktk "ifc/lib/cimgui.dll")) (:linux (asdf/system:system-relative-pathname :vktk "ifc/lib/cimgui.so")))

(cffi:use-foreign-library cimgui)

Message ID: @.***>

[richardjdare]

I've been playing with Kons-9 on Windows 10, SBCL 2.2.7 x86-64 If I minimize the app, or click its icon in the windows taskbar a couple of times to minimize/maximize it, I get a division by zero error and the window closes.

I think the problem is in window-size-callback in minimal-ui.lisp. The w and h parameters are zero in this situation.

I've not been able to reproduce the issue on Linux.

This is what I get in the repl when it happens. (The "Window-size: 0 0" is my own debug printout inside window-size-callback)

KONS-9> (run)
#<SIMPLE-TASKS:CALL-TASK :FUNC #<FUNCTION (LAMBDA () :IN RUN) {10055B39AB}> :STATUS :SCHEDULED {10060E27D3}>
Window-size: 0  0
Date/time: 2022-08-29-12:13An unhandled error condition has been signalled:
                              arithmetic error DIVISION-BY-ZERO signalled
Operation was (/ 0 0).

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1001DD80A3}>
0: (SB-KERNEL::INTEGER-/-INTEGER 0 0)
1: ((LAMBDA (KONS-9::WINDOW KONS-9::W KONS-9::H) :IN "d:/projects/quicklisp/local-projects/kons-9/src/graphics/glfw/minimal-ui.lisp") #.(SB-SYS:INT-SAP #X001212E0) 0 0)
2: ((LAMBDA (SB-ALIEN::ARGS-POINTER SB-ALIEN::RESULT-POINTER FUNCTION) :IN "d:/projects/quicklisp/local-projects/kons-9/src/graphics/glfw/minimal-ui.lisp") #<unavailable argument> #<unavailable argument> #<unavailable argument>)
3: ("foreign function: #x140035AE5")
4: ("foreign function: #x1400055CC")
5: (KONS-9:SHOW-WINDOW #<KONS-9::SCENE frame bounds: 0 240, current: 0  {1005738FB3}>)
6: ((:METHOD SIMPLE-TASKS:RUN-TASK (SIMPLE-TASKS:CALL-TASK)) #<SIMPLE-TASKS:CALL-TASK :FUNC #<FUNCTION (LAMBDA NIL :IN KONS-9:RUN) {1006FD369B}> :STATUS :RUNNING {10060E27D3}>) [fast-method]
7: ((:METHOD SIMPLE-TASKS:RUN-TASK :AROUND (SIMPLE-TASKS:TASK)) #<SIMPLE-TASKS:CALL-TASK :FUNC #<FUNCTION (LAMBDA NIL :IN KONS-9:RUN) {1006FD369B}> :STATUS :RUNNING {10060E27D3}>) [fast-method]
8: ((:METHOD SIMPLE-TASKS:START-RUNNER (SIMPLE-TASKS:QUEUED-RUNNER)) #<SIMPLE-TASKS:QUEUED-RUNNER :STATUS :RUNNING {100256BF63}>) [fast-method]
9: ((SB-PCL::EMF SIMPLE-TASKS:START-RUNNER) #<unused argument> #<unused argument> #<SIMPLE-TASKS:QUEUED-RUNNER :STATUS :RUNNING {100256BF63}>)
10: ((:METHOD SIMPLE-TASKS:START-RUNNER :AROUND (SIMPLE-TASKS:RUNNER)) #<SIMPLE-TASKS:QUEUED-RUNNER :STATUS :RUNNING {100256BF63}>) [fast-method]
11: ((LAMBDA NIL :IN TRIVIAL-MAIN-THREAD::RUNNER-STARTER))
12: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::%INTERRUPT-THREAD))
13: ((FLET SB-UNIX::INTERRUPTION :IN SB-THREAD::RUN-INTERRUPTION))
14: ((FLET SB-THREAD::EXEC :IN SB-SYS:INVOKE-INTERRUPTION))
15: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-SYS:INVOKE-INTERRUPTION))
16: (SB-SYS:INVOKE-INTERRUPTION #<FUNCTION (FLET SB-UNIX::INTERRUPTION :IN SB-THREAD::RUN-INTERRUPTION) {100001AE1B}>)
17: (SB-THREAD::RUN-INTERRUPTION #.(SB-SYS:INT-SAP #X005FE770))
18: ("foreign function: #x140035AE5")
19: ("foreign function: #x14000546A")

[JMC-design]

Awolven, what is a 'normal' glfw install? They only provide a zip and don't tell you were to install it.

[JMC-design]

Richard, that sounds more like a windows issue. Iconifying or minimizing a window in linux is a state, not a change in window size. Seems like an easy fix though.

[awolven]

they provide an installable executable

On Mon, Aug 29, 2022 at 10:33 AM Johannes Martinez Calzada < @.***> wrote:

Awolven, what is a 'normal' glfw install? They only provide a zip and don't tell you were to install it.

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/15#issuecomment-1230482078, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMJVWAER5SMNIY2E4DLV3TJ53ANCNFSM56KRLEEA . You are receiving this because you commented.Message ID: @.***>

[JMC-design]

Do you have link? I only see zips.

[richardjdare]

As per mikelevins request I checked out sbcl-win64-tests and worked through the document, run-sbcl-win64.md I was able to follow the document and run demo-kernel.lisp on Windows 10. There were a couple of things that slowed me down a bit though.

Setting up the glfw dll At first glance it's not clear what to do with that bit of code that adds glfw to cffi:foreign-library-directories.

We could maybe write something similar to this:

1: Install the 64-bit version of glfw 2: Open kons-9.asd and look for this code near the top:

#+(and windows x86-64) 
(eval-when (:compile-toplevel :load-toplevel :execute)
  (pushnew "c:/Program Files/glfw-3.3.8/lib/"
           cffi:*foreign-library-directories*))

3: modify this line, (pushnew "c:/Program Files/glfw-3.3.8/lib")

changing the path from "c:/Program Files/glfw-3.3.8/lib" to the location of the directory containing glfw.dll on your computer. ... Or something like that.

Modify the definition of print-scene-view-help It might be worth writing something like, "Modify the definition of print-scene-view-help (in src\graphics\glfw\minimal-ui.lisp) so people can quickly find it.

[mikelevins]

Like JMC, I don't see an executable installer from glfw.org for 64-bit Windows; I see only zips containing readme, license, docs, and DLL binaries.

So I followed the instructions I found here:

https://subscription.packtpub.com/book/application-development/9781782169727/1/ch01lvl1sec13/installing-the-glfw-library-in-windows#:~:text=Download%20the%20source%20package%20from,and%20open%20cmake%2Dgui%20.

...adjusting for the different version of glfw, and for the fact that I was using the 64-bit libraries. CFFI doesn't find that install, which is no surprise; I have to tell it where to look.

If there is indeed a Windows installer that installs glfw in such a way that CFFI will find it without manual intervention then I would very much like to know where to find it. I'll modify the instructions to refer to it and send Kaveh a pull request as soon as I've tested the results.

For what it's worth, I see that chocolatey has GLFW available, but it's version 3.0.4.2; my testing branch is using 3.3.8.

[mikelevins]

@richardjdare I modified the Windows instructions on the sbcl-win64-testing branch per your suggestions. Let me know what you think.

[awolven]

I gave in some post a way to derive the shared object directory from the asdf system:

(defmacro my-define-foreign-library (name-and-options &body pairs) `(cffi:define-foreign-library ,name-and-options ,@(mapcar (lambda (pair) (list (first pair) (eval (second pair)))) pairs)))

(my-define-foreign-library cimgui (:darwin (asdf/system:system-relative-pathname :vktk "ifc/lib/cimgui.dylib")) (:windows (asdf/system:system-relative-pathname :vktk "ifc/lib/cimgui.dll")) (:linux (asdf/system:system-relative-pathname :vktk "ifc/lib/cimgui.so")))

(cffi:use-foreign-library cimgui)

Certainly that is better than hard coding the path.

At some point in the past I was able to download a windows installable for glfw3.4.

On Tue, Aug 30, 2022 at 9:06 AM mikel evins @.***> wrote:

@richardjdare https://github.com/richardjdare I modified the Windows instructions on the sbcl-win64-testing branch per your suggestions. Let me know what you think.

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/15#issuecomment-1231718614, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMK7AMINBTQJFV4FYJTV3YIOBANCNFSM56KRLEEA . You are receiving this because you commented.Message ID: @.***>

[mikelevins]

@awolven okay; fair enough. I previously solved a slightly different problem with CFFI and dylibs on macOS in a way that was kind of similar in spirit to your suggestion. I was hoping to avoid similar hackery here, but such is life.

Also, it would have been awfully nice to find a windows installer for glfw, but again, that's the way it goes. I'll figure it out.

[richardjdare]

@richardjdare I modified the Windows instructions on the sbcl-win64-testing branch per your suggestions. Let me know what you think.

@mikelevins thanks, that's great.

[mikelevins]

I have a locally-working script that downloads and unpacks glfw 3.8.8 on Windows in a project-local lib directory that is gitignored (so we don't stuff binaries into the repo).

It's in the "SBCL-win64-tests" branch, but not thoroughly tested, and not yet using proper pathnames. When I've addressed those failings, I'll try it with updated code from main and if it looks like it's working, send a PR.

[mikelevins]

I haven't pushed the code that automates downloading and loading glfw because I've been sidetracked debugging some quicklisp/sbcl issues that seem to be related to recent releases of sbcl.

In case anyone else is working with sbcl and quicklisp on Windows 11, I'll briefly describe what I've observed:

I'm seeing several basic quicklisp libraries failing to load with sbcl versions 2.2.6, 2.2.7, and 2.2.8. The same libraries load without any trouble on sbcl 2.2.3. I've been careful to reinstall both sbcl and quicklisp clean for each test, and cleared out the asdf cache each time.

If anyone can confirm or deny similar behavior, let me hear from you. Otherwise, I'll investigate on my own.

I'd rather not push a Windows version of kons-9 that requires a specific version of sbcl, but I will if it looks like the only short-term way to make it available.

[JMC-design]

So you're compiling from source? Is there the same issue on win10? I've only tried the binary release 2.0.0. on 10, which is probably my last version of windows ever.

[awolven]

Mikel: Is it a problem which could be addressed by asking the sbcl implementers?

On Tue, Sep 6, 2022 at 8:05 PM Johannes Martinez Calzada < @.***> wrote:

So you're compiling from source? Is there the same issue on win10? I've only tried the binary release 2.0.0. on 10, which is probably my last version of windows ever.

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/15#issuecomment-1238795439, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMKXNQ3CYC6PYUY4TFLV47S7FANCNFSM56KRLEEA . You are receiving this because you were mentioned.Message ID: @.***>

[richardjdare]

I've not had any issues building on Windows 10. I just tried it on my work machine, with a fresh clone of kons-9 and a fresh install of sbcl 2.2.8 and quicklisp.

[mikelevins]

I need to investigate the sbcl/quicklisp issues further.

I have several Windows systems that I'm working with, including two that are Windows hardware and two that are VMWare VMs. One piece of hardware is running Windows 11 Pro; the remaining systems are running Windows 10 Pro.

All of the problems I've seen occurred on the Windows 11 Pro hardware after a fresh install of sbcl 2.2.8, and a fresh install of quicklisp. A fresh install of sbcl 2..2.7 and of 2.2.6 with fresh installs of quicklisp yield the same results (problems loading basic quicklisp packages--nothing specifically to do with kons-9), most commonly breaking on package locks, but disabling package locks leads to other breakages.

A fresh install of sbcl 2.2.3 and of quicklisp resolves all problems, and everything (including kons-9) works fine.

That's what I know, so far.

It's entirely possible that something is wrong on that specific machine (I haven't reproduced this issue yet on any of the others), but any conclusions I draw at the moment would be speculation. I'm busy preparing a demo for a funding review, but I'll investigate further as soon as I can.

[mikelevins]

Mikel: Is it a problem which could be addressed by asking the sbcl implementers? … On Tue, Sep 6, 2022 at 8:05 PM Johannes Martinez Calzada < @.***> wrote: So you're compiling from source? Is there the same issue on win10? I've only tried the binary release 2.0.0. on 10, which is probably my last version of windows ever.

I did install 2.2.8 by building from source in one test, but in the other cases I installed from binary installers available here:

https://sourceforge.net/projects/sbcl/files/sbcl/

I don't yet know what to ask the sbcl implementors; I just need to spend some time investigating first.

[mikelevins]

I've not had any issues building on Windows 10. I just tried it on my work machine, with a fresh clone of kons-9 and a fresh install of sbcl 2.2.8 and quicklisp.

That's useful information. I should test the hypothesis that it's somehow related with the Windows version (though that seems a little far-fetched to me). If anyone can build and test kons-9 with sbcl 2.2.8 on Windows 11 Pro, I'd like to know about it.

[JMC-design]

Doesn't sound far fetched. Windows 11 is pretty bad to play opengl games on. I think it's the first time Linux does a better job overall. I think at the very least we have to be religious about setting/unsetting state, because maybe win11 isn't so careful and leaks stuff, and since we leak stuff(change in shading being an example or has that been fixed?) together probably makes a bad fit.

[kaveh808]

The OpenGL shading change on window reopen has not been fixed.

[mikelevins]

I haven't looked at the Windows version of kons-9 in a few days because I needed to provide some help to the main collaborator on my other Lisp IDE project. We got his needs met, and I can pay attention to this again.

My collaborator works primarily on Windows. In the course of working on his requirements, I discovered additional evidence that something may be off with SBCL and quicklisp on recent Windows versions, for versions of SBCL from 2.2.6 and up. I don't know any details about what's going on, yet. I just know that I had problems loading quicklisp libraries on WIndows 11 with versions of SBCL later than 2.2.5, and I found a few complaints from other users of SBCL reporting similar problems (quicklisp failures occasioned by unexpected problems with package locks).

I need to investigate rurther. Investigation means installing different versions of SBCL on different versions of Windows (fortunately, I have machines on which I can do that), and nailing down where problems happen. It also probably means asking questions of the SBCL implementors, but I want to be able to ask precise questions, so I need to tinker some, first.

With luck, I can find an SBCL configuration that makes the problem disappear (I vaguely recall that SBCL has some variables for controlling how warnings are handled that may be of help, and I think its package-locks mechanism also provides some hooks).

Anyone with fresher knowledge of these matters is welcome to contribute guidance.

[richardjdare]

@mikelevins I got Windows 11 going in Virtual Box (running on Windows 10) to see if I could help. I was able to run sbcl-win64-tests in SBCL 2.2.8 without any problems. I was even able to run the main branch. The only issue I had was I had to use the mingw version of glfw3.dll. CFFI wouldn't load the other versions for some reason. (I usually use the msvc versions on Windows 10)

There may be something going on with the latest versions of SBCL though. I've had package lock issues with a few libraries myself.

[richardjdare]

@mikelevins The "Package Lock" problems I was having on Windows 11, SBCL 2.28 occurred in libraries that used usocket. It fails to compile when it attempts to use SB-ALIEN::HINSTANCE, and reports:

Lock on package SB-ALIEN violated when interning HINSTANCE while in package USOCKET.

There was a recent update to usocket that fixed this, but its not on Quicklisp yet. https://github.com/usocket/usocket/commit/341f06175431d822fd529ea8dfd283d5f0ef0243

SB-ALIEN::HINSTANCE was an internal symbol that got removed in 2.2.7 https://groups.google.com/g/sbcl-devel/c/UG3HcLfJuH0

Are these issues similar to the ones you're having?

In the end I tried loading over 300 popular packages on Windows 11 with SBCL 2.2.8. A few had problems for unrelated reasons but the usocket problem popped up quite often.

[mikelevins]

Yes, that sounds like exactly what I was seeing. THanks for this information; I'll look into it. It may suggest a straightforward solution (for example, I'll try pulling the latest usocket into local-projects and see where that gets me).

Thanks again! I'll report my results.

[mikelevins]

Thanks again! I'll report my results.

Results: cloning the HEAD of usocket into quicklisp/local-projects appears to fix the problem. I'm able to load kons-9 and run demo code.

I'll proceed with win64 integration tasks and report back if I run into another roadblock.

Thanks, @richardjdare!

Oh, and I should mention that I did my testing on Windows 11 Pro version 21H2, build 22000.978, with SBCL win-64 version 2.2.8.

Once the fixed usocket makes it into quicklisp, it should no longer be necessary to clone it into local-projects.

[kaveh808]

Hey guys,

I was wondering where this stands. Can I announce that we support Windows?

[awolven]

windows worked for me when I tried it, but that may have been a few commits ago.

On Fri, Oct 7, 2022 at 4:23 PM Kaveh Kardan @.***> wrote:

Hey guys,

I was wondering where this stands. Can I announce that we support Windows?

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/15#issuecomment-1272097870, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMKIDZGUSHVETI74FRLWCCIGLANCNFSM56KRLEEA . You are receiving this because you were mentioned.Message ID: @.***>

[richardjdare]

I've been testing the latest main branch, and also got the text-engine branch going on Windows 10. So far so good. I will run through more tests over the weekend and report back if I find anything.

[kaveh808]

Excellent, thanks @richardjdare

If you get a chance, grab the latest version. I fixed some bugs and there is an initial UI tutorial in demo-ui.lisp.

[maacl]

I have today downloaded and run master on Windows 11. After downloading glfw3.dll and placing in root of project it runs without any issues using SBCL 2.0.0 and the README.md instructions.

[kaveh808]

Excellent, @maacl. Thank you for testing this.

[mikelevins]

@kaveh808. @richardjdare, what do you guys think; can we consider that kons-9 works on WIndows? Can we close this issue as completed? If not, what further work should be done?

[maacl]

I have not seen any issues. Personally I would declare victory. Perhaps consider giving explicit instructions on which dll to get and where to place it?

On Fri, Nov 11, 2022 at 1:58 PM mikel evins @.***> wrote:

@kaveh808 https://github.com/kaveh808. @richardjdare https://github.com/richardjdare, what do you guys think; can we consider that kons-9 works on WIndows? Can we close this issue as completed? If not, what further work should be done?

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/15#issuecomment-1311669152, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPUXFAM7BI56V6OGTWCTDWHY7FZANCNFSM56KRLEEA . You are receiving this because you were mentioned.Message ID: @.***>

[mikelevins]

Good idea.

[richardjdare]

I've had no issues on Windows recently. Don't forget that there is a Windows-specific bugfix inside sbcl-win64-tests that hasn't been merged back into main yet. (see here for details https://github.com/kaveh808/kons-9/commit/159d7572847c5dff7500db36581cf1e78c3ef521)

[kaveh808]

@richardjdare I had missed that pull request. I'll do it by Monday.