Statically-linked SDL

[sherief]

Lite is a great editor and one of the rare programs that load super fast - would you consider statically linking SDL2 for the Windows build (and the CRT too, if that's not the case)? A single self-contained executable would be great to carry around and maybe even run from a remote server when I need to. The increased utility from this portability is worth it IMO and since the project is open source any issues that can be fixed with updating the SDL2 DLL can be done directly to the source / build.

[rxi]

Thanks for the kind words! I typically do static link on other projects for the reason you listed, of having a single executable -- for this particular project I didn't feel it made sense as you always need the data directory and all the files in it, so you're always having to deal with more than a single file anyway which somewhat invalidates the reasons for doing this.

[sherief]

ahem I was planning to take it and embed the other files inside the executable as resources, as described / done here: https://sherief.fyi/post/atomic-walk/ :D would that change your mind?

[1bardesign]

Just chiming in that I feel like half the appeal of lite is being able to tinker with most of the guts of the program by just modifying lua files, and that a resources-in-binary approach would be counter to that.

I support statically linking SDL though :wink:

[sherief]

I totally agree and I plan to mount the local directory in addition to the embedded resources so local files overwrite the ones in the executable. You can read my thoughts and the process at https://sherief.fyi/post/atomic-application/

I personally like people to play around with my apps and modify shaders to their heart’s content so I always mount the current working directory and also add a command line option to extract the built-in archive so someone who only has the app can run it using the command line option --extract [path] and it will extract the built-in archive to the specified path so the user can inspect and modify the files.

[MaD70]

@sherief: the approach you are suggesting is absolutely legit and tried and tested, see for example in the Tcl/Tk world Starkits and Starpacks.

Googling around I see that is somewhat popular even in the Lua world, see Bundle, Amalg, Squish and so on. Note in particular that (at least) Bundle has the upgradability feature that you mentioned in your last comment:

How it works

[...] This allows mixed deployments where some modules and assets are bundled inside the exe and some are left outside, with no changes to the code and no rebundling needed. External modules always take precedence over embedded ones, allowing partial upgrades to the original executable without the need for a rebuild. Finally, one of the modules (embedded or not) can be specified to run instead of the usual REPL, effectively enabling single-executable app deployment for pure Lua apps with no glue C code needed.

[MaD70]

I managed to statically link SDL2 in Lite. I'm on Windows 10 (64 bits) and I use msys2 to build (if I recall correctly, after installation a pacman -S base-devel gcc zip should be sufficient to build Lite). It's just a matter of adding some linker flags.

But first in build.sh I changed

cflags="-Wall -O3 -g -std=gnu11 -Isrc"

in

cflags="-Wall -Os -lto -fuse-linker-plugin -std=gnu11 -Isrc"

For -Os, -flto and -fuse-linker-plugin flags see GCC docs. This is not necessary but it results in a bit smaller executable and it starts a bit faster on my old machine.

Now the changes necessary to link SDL2 statically. Again in build.sh in the if that selects the platform

if [[ $* == *windows* ]]; then
  platform="windows"
  outfile="lite.exe"
  compiler="x86_64-w64-mingw32-gcc"
  cflags="$cflags -DLUA_USE_POPEN -Iwinlib/SDL2-2.0.10/x86_64-w64-mingw32/include"
  lflags="$lflags -Lwinlib/SDL2-2.0.10/x86_64-w64-mingw32/lib"
# ADD HERE
  lflags="-lmingw32 -lSDL2main $lflags -mwindows -o $outfile res.res"
  x86_64-w64-mingw32-windres res.rc -O coff -o res.res
else

I added the line:

  lflags="-static $lflags -lsetupapi -ldinput8 -ldxguid -ldxerr8 -lkernel32 \
          -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lversion \
          -luuid -static-libgcc"

I obtained an executables of 1'618'432 bytes.

In build_release.sh I commented what is not relevant for me (I don't cross-compile for other platforms):

#!/bin/bash
./build.sh release windows
#./build.sh release
rm lite.zip 2>/dev/null
#cp winlib/SDL2-2.0.10/x86_64-w64-mingw32/bin/SDL2.dll SDL2.dll
#strip lite
strip lite.exe
#strip SDL2.dll
#zip lite.zip lite lite.exe SDL2.dll data -r
zip lite.zip lite.exe data -r

Next thing I want to try is to compile SDL2 from sources with Link Time Optimizations (-flto -fuse-linker-plugin) like all the rest.

[MaD70]

I compiled SDL2 (2.0.12) with LTO and linked statically to Lite with this small addition to build.sh:

cflags="$cflags -fno-asynchronous-unwind-tables -fno-unwind-tables"

after the previous setting of cflags (see my previous comment).

The resulting executable is of 1'211'904 bytes and starts immediately after a double click in explorer on my 14 years old pc.

ADDENDUM: let me explain what I did with my little experiment, so that more people with different levels of experience can understand.

Not knowing SDL2 nor Lua, I checked what could be done with the least effort, just setting certain compiler/linker flags while building SDL2/Lua/lite.

Of course, even with LTO I was not expecting exceptional results (still a cut of a little less than 400 KiB it's not bad). In fact, if the Lua binding to SDL2 is complete, the Lua interpreter embedded in lite is "using" the entire SDL2 library, even if lite uses only part of SDL2 (no dead code elimination, there is no dead code).

Standardizing on a suitable subset of SDL2 that can be used by lite and its extensions could be conducive to better results in terms of executable size. SDL2 build can be customized to exclude a lot of subsystems (see ./configure --help in a bash shell, at the root directory of SDL2 sources). To avoid compilation errors, one just cut them out also from Lua binding (trimming only Lua binding to SDL should suffice, but compiling with LTO is somewhat slow, so configuring SDL2 to compile only the desired subsystems should speed up the process).

Candidate subsystems for exclusion could be, for example: audio, ~~video (wrong: I thought it was a subsystem for playing video files but it's the drawing subsystem), joystick, haptic, sensor, and so on.~~

Wrong hypothesis: it seems that there is no Lua binding to SDL2 in the source tree of lite, so SDL2 is called only from the C part of lite. But there is a flag to disable a "dynamic API" in SDL2. Recompiling SDL2 with this API disabled doesn't do much for trimming lite executable size (1'127'424 bytes a difference of about 480 KiB vs the non-LTO enabled recompiled SDL) but I think it makes it perform a little better (even in term of memory consumption).

[sherief]

The world needs more people like you, @MaD70

[MaD70]

@sherief: for what it's worth, I would like that more people were sensible to human factors as you seem to be. But after all these years the inmate are still running the asylum (yes, they really are) and are still producing coffeepots for masochists at an alarming rate.

[MaD70]

As a curiosity I compiled SDL2 disabling most subsystems with the following script. lite went down to 979 KiB (1'003'008 bytes).

In what follow <base dir> is the base directory where you are putting the directory SDL2 (see the directory tree in the script). Download SDL2-2.0.12 source code (.zip) and unzip it under SDL2.

First in <base dir>/SDL2/SDL2-2.0.12/src/dynapi/SDL_dynapi.h remember to change SDL_DYNAMIC_API from 1 to 0:

/* everyone else. This is where we turn on the API if nothing forced it off. */
#ifndef SDL_DYNAMIC_API
#define SDL_DYNAMIC_API 1
#endif

Before running it remember to replace <base dir> in the following script too (--prefix wants an absolute path).

#!/bin/bash
# ---------------------------------------------------------
# directory tree:
# SDL2 <─ put this script here, it must be already created
# ├ build <───┮ this dirs are created by the script
# ├ install <─┘
# └ SDL2-2.0.12 <─ created by unpacking SDL sources in SDL2
# ---------------------------------------------------------
mkdir build
mkdir install
cd build

export CFLAGS="-Os -flto -fuse-linker-plugin"

../SDL2-2.0.12/configure --prefix=<base dir>/SDL2/install --build=x86_64-w64-mingw32 \
  --disable-shared --disable-libc --disable-audio --disable-joystick --disable-haptic \
  --disable-sensor --disable-power --disable-timers --disable-oss --disable-alsa \
  --disable-alsatest --disable-alsa-shared --disable-jack --disable-jack-shared \
  --disable-esd --disable-esdtest --disable-esd-shared --disable-pulseaudio \
  --disable-pulseaudio-shared --disable-arts --disable-arts-shared --disable-nas \
  --disable-nas-shared --disable-sndio --disable-sndio-shared --disable-fusionsound \
  --disable-fusionsound-shared --disable-diskaudio --disable-dummyaudio \
  --disable-libsamplerate --disable-libsamplerate-shared --disable-video-vulkan \
  --disable-libudev --disable-dbus --disable-ime --disable-ibus --disable-fcitx \
  --disable-input-tslib --disable-pthreads --disable-pthread-sem --disable-sdl-dlopen \
  --disable-hidapi --disable-clock_gettime --disable-rpath

make
make install

When it finishes, create a directory SDL2-2.0.12-lto under winlib in the directory of lite source code, then copy into it the content of directory <base dir>/SDL2/install. Update the build.sh script that compile lite accordingly:

  cflags="$cflags -DLUA_USE_POPEN -Iwinlib/SDL2-2.0.12-lto/include"
  lflags="$lflags -Lwinlib/SDL2-2.0.12-lto/lib"

Run it and it should produce a leaner lite.

[sigzero]

I get the error in the attached screenshot on Windows 10 for both 1.11 and 1.10, Am I missing something?