emscripten-core / emsdk

Emscripten SDK
http://emscripten.org
Other
2.99k stars 682 forks source link

Bazel: @emsdk cannot pick up SDL2 headers and libraries. #1099

Closed sohanjg closed 3 months ago

sohanjg commented 2 years ago

Bazel build with emsdk (for wasm_cc_binary) is unable to fetch and discover the SDL2 file on specifying them in linkopts ("-sUSE_SDL=2", "-sUSE_SDL_IMAGE=2", "-sUSE_SDL_TTF=2", "-sUSE_SDL_NET=2") and coopts ("-DUSE_SDL=2", "-DUSE_SDL_IMAGE=2", "-DUSE_SDL_TTF=2", "-DUSE_SDL_NET=2",) as mentioned in https://emscripten.org/docs/compiling/Building-Projects.html?#emscripten-ports Compilation fails with "SDL2/SDL.h" headers not found errors.

Note : SDL1 headers and libs/API, work out of the box. We are experimenting in Skia bazel environment.

sbc100 commented 2 years ago

The -s options are needed at compile time too. For example, you should use -sUSE_SDL=2 and not -DUSE_SDL2

sohanjg commented 2 years ago

Thanks ! Now it fails with some permission error, about RO file system. I am on Ubuntu 22.04 with sudo access. Pasting the build log.

`Starting local Bazel server and connecting to it... INFO: Analyzed target //example:hello_world_wasm (133 packages loaded, 12128 targets configured). INFO: Found 1 target... ERROR: /home/ubuntu/skia/tools/timer/BUILD.bazel:7:16: Compiling tools/timer/Timer.cpp failed: (Exit 1): emcc.sh failed: error executing command external/emsdk/emscripten_toolchain/emcc.sh '--sysroot=external/emscripten_bin_linux/emscripten/cache/sysroot' -fdiagnostics-color -fno-exceptions -fno-strict-aliasing -funsigned-char ... (remaining 155 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging Traceback (most recent call last): File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/4/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 4010, in sys.exit(main(sys.argv)) File "/usr/lib/python3.10/contextlib.py", line 79, in inner return func(*args, *kwds) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/4/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 4003, in main ret = run(args) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/4/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 1137, in run linker_inputs = phase_compile_inputs(options, state, newargs, input_files) File "/usr/lib/python3.10/contextlib.py", line 79, in inner return func(args, **kwds) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/4/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 2697, in phase_compile_inputs compile_source_file(i, input_file) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/4/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 2677, in compile_source_file cmd = get_clang_command(input_file) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/4/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 2618, in get_clang_command return get_compiler(src_file) + get_cflags(state.orig_args) + compile_args + [src_file] File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/4/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 889, in get_cflags ports.add_cflags(cflags, settings) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/ports/init.py", line 364, in add_cflags port.get(Ports, settings, shared) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/ports/freetype.py", line 19, in get ports.fetch_project('freetype', 'https://github.com/emscripten-ports/FreeType/archive/' + TAG + '.zip', 'FreeType-' + TAG, sha512hash=HASH) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/ports/init.py", line 159, in fetch_project fullname = os.path.join(Ports.get_dir(), name) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/ports/init.py", line 140, in get_dir shared.safe_ensure_dirs(dirname) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/utils.py", line 28, in safe_ensure_dirs os.makedirs(dirname, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 30] Read-only file system: '/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/cache/ports' Target //example:hello_world_wasm failed to build `

sbc100 commented 2 years ago

I think this is a limitation of the bazel setup for emsdk. IIRC you can't build new ports that were not already built when the emsdk was created/built. Is that right @walkingeyerobot ?

I though that we set FROZEN_CACHE which would make the error message a little more readable I think.

sohanjg commented 2 years ago

Thanks ! Is there any workaround for the limitation in bazel setup ? If not, does emsdk plan to upgrade built-in SDL1 to SDL2 anytime soon ?

sbc100 commented 2 years ago

I don't know of any way to add new ports to the bazel SDK today. Presumably the right solution would be to convert the ports builds into bazel BUILD files? Perhaps this could be done automatically, but I don't know of anyone that is working on it.

walkingeyerobot commented 2 years ago

The right solution is indeed to convert the ports builds into bazel BUILD files. This cannot be done automatically, but I am actively working on it. It's quite a large undertaking though, and I don't expect to finish for some time (on the order of months).

I'm not exactly sure how to workaround, but unsetting FROZEN_CACHE in bazel/emscripten_toolchain/emscripten_config would be my first stop. I don't expect this to immediately work, but the error message that gives would help inform the next step. You may also be able to build the cache outside of bazel and then copy the appropriate files to wherever bazel is looking for them.

sbc100 commented 2 years ago

Can we automatically generate BUILD files from tools/system_libc.py? I would have through that would be the simplest way to keep them up-to-date.. I was thinking of generating ninja or cmake files from that code too.. in order to speed up out builds.

walkingeyerobot commented 2 years ago

Yes, we can generate most of the rules required. However, there's some logic which will make it tricky to generate (i.e. switching what files are part of the input based on certain features such as threading). Generating these rules seems much preferable to keeping them up to date by hand. Also, there's a lot of infrastructure that needs to be figured out and written to make sure the new library rules are built and integrated correctly.

sohanjg commented 2 years ago

Thank you for checking this. With FROZEN_CACHE as None/False, doesn't give us much more info except that it fails while creating directory because of permission issues.

`ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/spirv_cross/BUILD.bazel:4:11: Compiling spirv_cross_parsed_ir.cpp failed: (Exit 1): emcc.sh failed: error executing command external/emsdk/emscripten_toolchain/emcc.sh '--sysroot=external/emscripten_bin_linux/emscripten/cache/sysroot' -fdiagnostics-color -fno-exceptions -fno-strict-aliasing -funsigned-char ... (remaining 33 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging Traceback (most recent call last): File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/9/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 40, in import emscripten File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/emscripten.py", line 23, in from tools import building File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/building.py", line 21, in from . import shared File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/shared.py", line 714, in Cache = cache.Cache(config.CACHE) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/cache.py", line 35, in init self.ensure() File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/cache.py", line 81, in ensure utils.safe_ensure_dirs(self.dirname) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/utils.py", line 28, in safe_ensure_dirs os.makedirs(dirname, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 30] Read-only file system: '/home/ubuntu/.emscripten_cache' ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/spirv_cross/BUILD.bazel:4:11: Compiling spirv_cross.cpp failed: (Exit 1): emcc.sh failed: error executing command external/emsdk/emscripten_toolchain/emcc.sh '--sysroot=external/emscripten_bin_linux/emscripten/cache/sysroot' -fdiagnostics-color -fno-exceptions -fno-strict-aliasing -funsigned-char ... (remaining 33 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging Traceback (most recent call last): File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/sandbox/linux-sandbox/10/execroot/skia/external/emscripten_bin_linux/emscripten/emcc.py", line 40, in import emscripten File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/emscripten.py", line 23, in from tools import building File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/building.py", line 21, in from . import shared File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/shared.py", line 714, in Cache = cache.Cache(config.CACHE) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/cache.py", line 35, in init self.ensure() File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/cache.py", line 81, in ensure utils.safe_ensure_dirs(self.dirname) File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/756dee633784ef959e49802380c6f32b/external/emscripten_bin_linux/emscripten/tools/utils.py", line 28, in safe_ensure_dirs os.makedirs(dirname, exist_ok=True) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode) OSError: [Errno 30] Read-only file system: '/home/ubuntu/.emscripten_cache' `

achoum commented 1 year ago

Hi @sohanjg,

I am currently trying to compile a bazel+emscripten+sdl2 binary and hitting some issues (https://github.com/emscripten-core/emsdk/issues/1229), and I saw your bug :).

Have you been able to solve your situation and compile your project with bazel+emscripten+sdl2?

Cheers,

sohanjg commented 1 year ago

Hey ! No, we went for a workaround and included sdl2 as a separate 3rd party library in our project, instead of relying on emscripten. Here is a WIP version of it, if it helps, https://skia-review.googlesource.com/c/skia/+/573643

achoum commented 1 year ago

Sorry to hear about the workaround, and thanks a lot for the change link :). This is a great help.