Manual Build process for Emscripten and Mobile

[frink]

I'd like to be able to build without fips for Emscripten and the Mobile platforms. But fips is so tightly integrated that I'm finding it hard to do a basic Makefiles. Could you add this to the documentation please?

[floooh]

I added some "build-system-agnostic" instructions to the readme a little while ago here (but I just noticed that I missed emscripten there):

https://github.com/floooh/sokol-samples#how-to-build-without-a-build-system

The only two reasons why this is a bit more tricky with the sokol-app samples are:

• some sokol headers must be compiled as Objective-C (on macOS and iOS) or C++ (on UWP), so for the sokol-app samples the sokol header implementations are compiled into a separate library instead of being included right into the sample source code
• the integration of the shader compiler is very build-system specific

For Android the whole process of setting up the Android SDK/NDK and then actually building APKs without a build system is so broken by design that I'd rather skip any instructions (e.g. look at those two python scripts to get an idea what the instructions would need to cover):

• https://github.com/floooh/fips/blob/master/mod/android.py
• https://github.com/floooh/fips/blob/master/tools/android-create-apk.py

I will add a section for building the emscripten samples (under https://github.com/floooh/sokol-samples/tree/master/html5) though, these will all be one-liners once the emscripten SDK has been installed.

[frink]

Cross platform is never as simple as it could be...

It probably makes sense to to put up a YMMV notice on the mobile platforms and just pointing to the build scripts like you did for me is enough. Are there weird scripts for iOS as well?

P.S. - There may be a more elegant way to do things for Android without the whole SDK/NDK. I was just reading this morning in a PR for Dear Imgui (PR:3446) using Gradle... It looked like a lot less gymnastics than using the full SDK/NDK. Seemed promising...

[floooh]

Gradle is just the current "Android build system flavour of the month", this is exactly one of the fundamental problems of the whole Android development workflow: they change build system details all the time, so that nothing works for longer than half a year or so. Using only cmake and calling the underlying tools directly to build the final APK (this is the only part that Gradle actually simplifies) is actually more stable and easier to maintain.

...what the Android SDK would actually need is a compiler/linker wrapper like emcc in the emscripten SDK, which would directly generate ready-to-use APKs the same way that emcc can generate ready-to-use HTMLs, unfortunately this seems to be a too obvious solution for the NDK team to consider ;)

[frink]

I'm new to Android development. Would love to see less build tools used in most projects I work on. I feel Like I spend more time debugging Python and Node than I do writing C sometimes. I wish there was a simple build system that just worked without external dependencies. I've read a few who claim to be. But the tooling is getting ridiculous these days!

[cshenton]

I'll echo this but specific for a sapp application targeting emscripten. Even a pointer to the relevant part of the fips codebase would be a huge helper 👍

[frink]

@cshenton see: https://github.com/floooh/fips/blob/master/cmake-toolchains/emscripten.toolchain.cmake

this seems to be where a lot of the magic config happens...

[frink]

fips is python + cmake

[floooh]

@frink @cshenton the emscripten.toolchain.cmake is a bit overkill, the emscripten-specific samples under

https://github.com/floooh/sokol-samples/tree/master/html5

can actually be built with one-liners, I just need to add a section to the README for this. For instance:

> cd sokol-samples/html5
> emcc clear.c -o clear.html -I [path-to-sokol-headers-dir]
> emrun clear.html

The only reasons that the cross-platform sokol_app.h samples under https://github.com/floooh/sokol-samples are a bit more complicated to build are

1. the shader cross-compilation via sokol-shdc (which is integrated into the build process, but could also run manually)
2. some samples have other dependencies, like Dear ImGui, libmodplug etc...

[frink]

But the html5 samples also include things like ImGui? (e.g. html5/imgui-emsc.cc)

So other than using sokol_app and sokol_imgui providing the scaffolding is there much other diffident between html5/imgui-emsc.cc and sapp/imgui-sapp.cc? The ImGui guts seem nearly identical...

Right now I'm hacking on the samples project adding my own stuff. But I'd really like to remove everything else and have a template project without a hard fips dependencies.

[floooh]

is there much other diffident between html5/imgui-emsc.cc and sapp/imgui-sapp.cc

The platform-specific samples in the d3d11/html5/glfw/metal... directories are essentially tests and examples of using sokol_gfx.h without sokol_app.h. The application-wrapper code (e.g. here https://github.com/floooh/sokol-samples/blob/master/html5/emsc.h, or here https://github.com/floooh/sokol-samples/blob/master/d3d11/d3d11entry.c) is much more minimal and "hacky" than what's in sokol_app.h (or SDL / GLFW), but the actual sokol_gfx.h code is mostly identical to the samples in the sapp directory.

[frink]

Could you for example use:

cd sokol-samples/html5
emcc cimgui-sapp.c [cimgui files] -o cimgui.html -I [path-to-sokol-headers-dir] -I [path-to-cimgui]

In other words, is there any other craziness besides just mapping which files and headers get included?

[floooh]

Yeah basically, in the case of the imgui samples this works because sokol_imgui.h comes with embedded shader code (e.g. sokol-shdc doesn't need to run).

The 3D backend to use isn't autodetected in the source code and must be defined via -D SOKOL_GLES2, but other then that it's just a list of source files and header search paths.

Let's use the imgui-sapp samples, because that requires fewer sources than cimgui-sapp:

> cd sokol-samples/sapp
> emcc imgui-sapp.cc ../libs/sokol/sokol.c ../../fips-imgui/imgui/imgui.cpp ../../fips-imgui/imgui/imgui_demo.cpp ../../fips-imgui/imgui/imgui_draw.cpp ../../fips-imgui/imgui/imgui_widgets.cpp -I../../sokol -I../../sokol/util -I../../fips-imgui/imgui -D SOKOL_GLES2 -o imgui-sapp.html -O3

Or split over several lines for better readability:

emcc imgui-sapp.cc 
../libs/sokol/sokol.c
../../fips-imgui/imgui/imgui.cpp
../../fips-imgui/imgui/imgui_demo.cpp
../../fips-imgui/imgui/imgui_draw.cpp
../../fips-imgui/imgui/imgui_widgets.cpp
-I../../sokol
-I../../sokol/util
-I../../fips-imgui/imgui
-D SOKOL_GLES2
-o imgui-sapp.html
-O3

PS: compiling like this gives you emscripten's default "shell.html" file, it works but looks ugly. To get the WebGL canvas stretched over the entire browser window, add this command line option too:

--shell-file ../webpage/shell.html

[frink]

Thanks for the explanation...

imgui-sapp [...] requires fewer sources than cimgui-sapp...

Assuming cimgui is more difficult because you need both cimgui and imgui sources to compile...?

sokol_imgui.h` comes with embedded shader code [so] sokol-shdc doesn't need to run.

I guess I should next ask about sokol-shdc. Maybe we can pick on an easy example like quad-sapp.glsl...

cd sapp/
sokol-shdc --input quad-sapp.glsl --output quad-sapp.glsl.h --slang=glsl300es

Is that on the right track?

[floooh]

Assuming cimgui is more difficult because you need both cimgui and imgui sources to compile...?

Yes, cimgui is just a C wrapper around ImGui, so you need to add both the cimgui sources and the imgui sources to the compilation.

sokol-shdc --input quad-sapp.glsl --output quad-sapp.glsl.h --slang=glsl300es

Yes. The exact arguments used for the sokol-samples are a bit different, but the above should also work. For reference:

https://github.com/floooh/sokol-tools-bin/blob/d4c5bff725fdac385a04df31846a202917511644/fips-files/generators/SokolShader.py#L39-L46

...also see the documentation:

https://github.com/floooh/sokol-tools/blob/master/docs/sokol-shdc.md

[frink]

Wow digging deeper into this it has problems because installing Emscripten itself is Royal Pain. I ended up adding it as a submodule. Really close to having a Makefile building an app.

The wasm and html generated by my scripts are roughly the same size as the stuff gneerated by fips but my JS file is 121K whereas yours is only 32K from the same sapp file. (I've been modifying the sapp/cimgui_sapp.c file up until now...) Are there some other settings that you're calling to shrink the size of the JS?

[floooh]

Wow digging deeper into this it has problems because installing Emscripten itself is Royal Pain.

Did you follow the emsdk installations instructions here?

https://emscripten.org/docs/getting_started/downloads.html#installation-instructions

I didn't have any trouble with emsdk for a long time, especially compared to "pre-emsdk" times.

JS file is 121K whereas yours is only 32K from the same sapp file

Fips has a lot of tweaks in its emscripten-cmake toolchain file to balance size vs performance (like switching off the filesystem emulation layer, and using the small-but-slow emmalloc instead of the vanilla jemalloc):

https://github.com/floooh/fips/blob/2935b9674d45c3f69eb72cbb9d56b62ca5bfe698/cmake-toolchains/emscripten.toolchain.cmake#L18-L38

[frink]

Found another issue in trying to work through this...

For my test app, I basically combined the quad-sapp background with the cimgui-sapp code to get a nice background behind my ImGui App. But then I started themeing and realized that the fading background color might change so I call sg_update_buffer() in the frame callback to make sure that we have the updated color in the background. But when I build this it fails a assertion somewhere before it renders the first frame... (The code works fine when built with fips...)

The 3D backend to use isn't autodetected in the source code and must be defined via -D SOKOL_GLES2, but other then that it's just a list of source files and header search paths.

I noticed buried in the CMake/fips stuff that you are actually using -DSOKOL_GLES3 rather than -DSOKOL_GLES2... Building with -DSOKOL_GLES2 generally works except for the blows up when I call sg_update_buffer() claiming an assertion failed. Building with with -DSOKOL_GLES3 it complains about a handful of missing symbols. It looks like maybe you are linking to GLFW because it has those missing symbols. (Hope I'm reading the CMake and fips code right...)

Is GLFW a dependency when running SOKOL_GLES3?

Also wondering if there's a way to get a call stack to know where it's failing? This is probably an Emscripten thing. But haven't found the flags that emcc needs yet.

[frink]

Just to make sure it wasn't my code I tried copying the quad_sapp.c example. I modified it with the two offending lines.

float vertices[] = {
    // positions                    colors
    -1.0f,  1.0f, 1.0f,      0.5f, 0.0f, 0.4f, 1.0f,
    1.0f,  1.0f, 1.0f,      0.4f, 0.0f, 0.3f, 1.0f,
    1.0f, -1.0f, 1.0f,      0.1f, 0.0f, 0.2f, 1.0f,
    -1.0f, -1.0f, 1.0f,      0.2f, 0.0f, 0.1f, 1.0f,
};
sg_update_buffer(iapp_gconf.bind.vertex_buffers[0], vertices, sizeof(vertices));

Building with emcc -DSOKOL_GLES3 results in these errors...

error: undefined symbol: glBlitFramebuffer (referenced by top-level compiled C/C++ code) warning: Link with -s LLD_REPORT_UNDEFINED to get more information on undefined symbols warning: To disable errors for undefined symbols use -s ERROR_ON_UNDEFINED_SYMBOLS=0 warning: _glBlitFramebuffer may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library error: undefined symbol: glGetStringi (referenced by top-level compiled C/C++ code) warning: _glGetStringi may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library error: undefined symbol: glReadBuffer (referenced by top-level compiled C/C++ code) warning: _glReadBuffer may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library ...

Aren't these symbols from GLFW?

If built with emcc -DSOKOL_GLES2 it builds fine and blows up like mentioned above. Identical code built by fips doesn't blow up. Wading though fips I'm not really sure what you'r linking against. But I'm missing something...

[frink]

Okay so I found what the symbol problem was...

When building -DSOKOL_GLES3 you need to add -DUSE_WEBGL2 to keep the symbols right.

Sadly this still doesn't help my assert problem yet.

I also notice that the fips build uses the flag FIPS_NO_ASSERTS_IN_RELEASE in the yaml configs for emscripten. fips.cmake claims that it "Remove asserts in release-mode". However, I cannot find where that actually does anything other than set a flag that is never referenced. If this sets a flag for emcc I didn't find it.

[floooh]

These seem like several unrelated problems:

sg_update_buffer() claiming an assertion failed

It would be important to know what the assertion says to know what the error is. E.g. the update-data might be too big to fit into the buffer, or sg_update_buffer() might have been called more then once per frame on the same buffer.

Is GLFW a dependency when running SOKOL_GLES3?

No, GLFW is only needed for the samples that use GLFW instead of sokol_app.h for window management:

https://github.com/floooh/sokol-samples/tree/master/glfw

-DSOKOL_GLES2 generally works except for the blows up when I call sg_update_buffer()

This is really weird, those two things should not be related. In general, if you only need GLES2/WebGL functionality, you should compile with SOKOL_GLES2, and only use SOKOL_GLES3 if you need GLES3/WebGL2 features. For instance the emscripten-specific samples in here either use SOKOL_GLES2 or SOKOL_GLES3: https://github.com/floooh/sokol-samples/tree/master/html5

...Aren't these symbols from GLFW?

No, these are regular desktop-GL and GLES3 functions and called in the non-GLES2 code paths in sokol_gfx.h

Also wondering if there's a way to get a call stack to know where it's failing?

If you compile with debug information (-g), then you'll get callstacks in the browser's Javascript console.

Sadly this still doesn't help my assert problem yet.

What does the assertion say on the Javascript console?

...FIPS_NO_ASSERTS_IN_RELEASE...

Ooops, yeah good find, this is sort-of deprecated, I should remove this from the build-config-files. I used this here in Oryol:

https://github.com/floooh/oryol/blob/eb08cffe1b1cb6b05ed14ec692bca9372cef064e/fips-files/include.cmake#L182-L184

...this was because Oryol defauls to keeping assert even in release mode. But with sokol I have switched to using the standard assert() macros which are controlled by the NDEBUG define.

[frink]

Thanks for working with me in all this. Once I have this working good I'll contribute a PR on manually building with Emscripten to the README.md.

These seem like several unrelated problems

Agreed. I solved the GLES3 problem. Now working on the sg_update_buffer() problem. Will follow your suggestion bellow and get back to you.

It would be important to know what the assertion says to know what the error is. If you compile with debug information (-g), then you'll get callstacks in the browser's Javascript console.

I'll compile with this and see if I can get further down the road...

...Aren't these symbols from GLFW? No, these are regular desktop-GL and GLES3 functions and called in the non-GLES2 code paths in sokol_gfx.h

I answered my own question on this. You have to use two flags to get GLES working.

When building -DSOKOL_GLES3 you need to add -DUSE_WEBGL2 to keep the symbols right.

I wonder if we should add an #ifdef in the relevant headers on this to avoid ambiguity of the two flags. Seems like this will be a common gotcha. But then again when WebGPU becomes standard that may throw a wrench in stuff again. But the two flags is definitely unclear...

...you should compile with SOKOL_GLES2, and only use SOKOL_GLES3 if you need GLES3/WebGL2 features.

For now I'm trying to get binary consistency with the fips examples so I am purposely trying to use the exact same flags as fips to prove that I'm generating the same binary with either build system. This is the only way to prove that I'm doing a perfect manual build.

Once I can get this level of consistency I will distill it all down and write a concise README section on building Emscripten manually. Like you said above, it would be good to include this... I'll send a PR with a good write up once I actually understand the process.

[frink]

Debug output from running with -g:

quad-sapp.js:1408 Uncaught RuntimeError: abort(Assertion failed: 0, at: lib/sokol/sokol_gfx.h,13051,_sg_validate_end) at Error at jsStackTrace (http://localhost/wasm/quad-sapp.js:1709:19) at stackTrace (http://localhost/wasm/quad-sapp.js:1726:16) at abort (http://localhost/wasm/quad-sapp.js:1402:44) at _assert_fail (http://localhost/wasm/quad-sapp.js:1732:7) at frame (http://localhost/wasm/quad-sapp.wasm:wasm-function[130]:0x5c41) at _sapp_emscframe (http://localhost/wasm/quad-sapp.wasm:wasm-function[140]:0xba44) at tick (http://localhost/wasm/quad-sapp.js:1879:30) at abort (http://localhost/wasm/quad-sapp.js:1408:11) at assert_fail (http://localhost/wasm/quad-sapp.js:1732:7) at frame (http://localhost/wasm/quad-sapp.wasm:wasm-function[130]:0x5c41) at _sapp_emsc_frame (http://localhost/wasm/quad-sapp.wasm:wasm-function[140]:0xba44) at tick (http://localhost/wasm/quad-sapp.js:1879:30)

Here is my frame() function:

void frame(void) {
        sg_begin_default_pass(&state.pass_action, sapp_width(), sapp_height());
        float vertices[] = {
                // positions                    colors
                -1.0f,  1.0f, 1.0f,      0.5f, 0.0f, 0.4f, 1.0f,
                 1.0f,  1.0f, 1.0f,      0.4f, 0.0f, 0.3f, 1.0f,
                 1.0f, -1.0f, 1.0f,      0.1f, 0.0f, 0.2f, 1.0f,
                -1.0f, -1.0f, 1.0f,      0.2f, 0.0f, 0.1f, 1.0f,
        };
        sg_update_buffer(state.bind.vertex_buffers[0], vertices, sizeof(vertices));
        sg_apply_pipeline(state.pip);
        sg_apply_bindings(&state.bind);
        sg_draw(0, 6, 1);
        __dbgui_draw();
        sg_end_pass();
        sg_commit();
}

Judging from above stack trace it looks like maybe something got in-lined.

I'm finding several calls to _sg_validate_end():

• _sg_validate_update_buffer()
• _sg_validate_apply_pipeline()
• _sg_validate_apply_bindings()

I'm guessing the first but I don't know why it would work in your fips compiler and not in mine unless fips sets -DSOKOL_VALIDATE_NON_FATAL. That's the only thing I can think of...

I tested building with -DSOKOL_VALIDATE_NON_FATAL and this compiles but generates a different error: sg_update_buffer: cannot update immutable buffer. Yet this compiles fine with fips.

[frink]

Is there no way to get fips and CMake to show their full calls so I can just grab the compiler flags from there? (I'm running with fips set config sapp-webgl2-wasm-ninja-release)

UPDATE: For fips + ninja to show a verbose build you just use: ./fips build -- -v

[frink]

Found it!!!

I was missing -DNDEBUG...

I also found that these settings gave me a 32% file size reduction in the wasm file.

-fstrict-aliasing -Wall -Wextra -Wno-multichar -Wno-unknown-pragmas -Wno-ignored-qualifiers -Wno-long-long -Wno-overloaded-virtual -Wno-deprecated-writable-strings -Wno-unused-volatile-lvalue -Wno-warn-absolute-paths -Wno-expansion-to-defined -Wno-missing-field-initializers

I'm now generating the same file sizes as fips. I've only got 34 flags passed to emcc. hehehe

But it works!

[lobotony]

This thread is immensely helpful, thank you. A quick heads up: I was trying to create a simple emscripten build setup myself today, and ended up using -s LLD_REPORT_UNDEFINED to better see missing symbols, just like emscripten suggested.

In the end, after seemingly satisfying every dependency, the build was still failing, and I kept seeing these:

wasm-ld: error: symbol exported via --export not found: __start_em_asm
wasm-ld: error: symbol exported via --export not found: __stop_em_asm

Turns out they disappear once you remove -s LLD_REPORT_UNDEFINED, and everything builds just fine.

[floooh]

@lobotony I'm not sure where those symbols are coming from TBH, haven't seen those before, but I haven't used LLD_REPORT_UNDEFINED so far.

There's this Emscripten-specific linker setting:

-s ERROR_ON_UNDEFINED_SYMBOLS=1

Also see here:

https://github.com/emscripten-core/emscripten/blob/9d385d5242eccdbba7220915d509f64f748a898f/src/settings.js#L1053-L1070

And maybe this projects also helps, it's using a plain cmake file, but I tried to keep this as simple as possible:

https://github.com/floooh/pacman.c