emscripten / WebAssembly port

[andreamancuso]

Hi,

I have been trying to compile maplibre-native using emscripten.

I can't figure out how to run emcmake in a way that does not cause errors.

I have created a new discussion in the emscripten github space: https://github.com/emscripten-core/emscripten/discussions/22146

I shall update this issue accordingly.

(P.S. you can track my PR: https://github.com/andreamancuso/react-wasm/pull/37 )

[louwers]

I am assuming you are setting CXX somewhere? I'll try it as well.

[andreamancuso]

I am assuming you are setting CXX somewhere? I'll try it as well.

Yes, I mean, I'm using the original CMakeLists.txt file

[louwers]

How does it choose emscripten?

[andreamancuso]

On Windows one would have to run a Visual Studio 2022 developer prompt, then run emsdk_env.bat from within the emsdk folder (once emsdk has been installed and activated).

On Linux one would have to install cmake and ninja then, from a terminal, run emsdk_env.sh from within the emsdk folder (once emsdk has been installed and activated).

I added a reference to emscripten's toolchain: set(CMAKE_TOOLCHAIN_FILE $ENV{EMSDK_HOME}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake).

Having said that, I now have some doubts as to whether this is using em++.

[andreamancuso]

I'm being silly - apologies.

I forgot I need to follow these instructions: https://emscripten.org/docs/compiling/Building-Projects.html#integrating-with-a-build-system

[andreamancuso]

I edited the description of the issue as I now I realized I never actually managed to compile maplibre-native using emscripten - it was just Visual Studio 2022 🤦

Sorry, I'll try harder.

[ntadej]

No problem! What you could do is to check how Qt toolchain file is defined for emscripten and base on that.

[louwers]

I already was surpised when you said you compiled libuv for WASM. ;-) Keep us posted!

[andreamancuso]

Aside from making myself look a plum, at least I think I have formulated the right question in the emscripten github project's discussion. Hopefully they'll be able to shed some light.

[andreamancuso]

I already was surpised when you said you compiled libuv for WASM. ;-) Keep us posted!

Well, I didn't. VCPKG (allegedly) did when I added it as a dependency during my previous attempts. The target I used is wasm32-emscripten.

[andreamancuso]

I am now trying a different approach, I added add_subdirectory(${DEPS}/maplibre-native) to my CMakeLists.txt so now I am certain that em++ is being used.

Also:

target_link_libraries(reactDearImgui PRIVATE imgui::imgui implot::implot nlohmann_json::nlohmann_json $<IF:$<TARGET_EXISTS:libuv::uv_a>,libuv::uv_a,libuv::uv> JPEG::JPEG mbgl-core) (notice how I am referencing libuv)

[andreamancuso]

@louwers https://github.com/emscripten-core/emscripten/discussions/22146#discussioncomment-9885793

I'll try disabling the warning

[andreamancuso]

Warnings disabled, I am now getting

In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/include\mbgl/gfx/context.hpp:7:
C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/src\mbgl/gfx/program.hpp:38:40: error: no type named 'UniformList' in 'mbgl::HeatmapProgram'
   38 |     using UniformList = typename Name::UniformList;
      |                         ~~~~~~~~~~~~~~~^~~~~~~~~~~
C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/src\mbgl/gfx/program.hpp:39:40: error: no type named 'TextureList' in 'mbgl::HeatmapProgram'
   39 |     using TextureList = typename Name::TextureList;
      |                         ~~~~~~~~~~~~~~~^~~~~~~~~~~

I don't get these errors when compiling using VS 2022. Seems like emscripten is getting confused somewhere, or maybe I shouldn't use C++23?

[andreamancuso]

I also tried importing https://github.com/mapbox/mapbox-gl-native using add_subdirectory() and got even more errors related to using std::variant.

[louwers]

We're compiling with C++17 right now.

[andreamancuso]

For some reason emscripten cannot resolve Name::AttributeList, Name::UniformList, Name::TextureList:

No idea why at the moment.

VS Code doesn't have a problem with that.

[andreamancuso]

I temporarily commented out those 3 'offending' types just so I can move forward. I came across this:

C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/src/mbgl/util/mapbox.cpp:97:56: error: no member named 'inserter' in namespace 'std'
   97 |     std::copy(tokenList.begin(), tokenList.end(), std::inserter(tokenMap, tokenMap.begin()));
      |                                                   ~~~~~^
1 error generated.

https://en.cppreference.com/w/cpp/iterator/inserter

I'm using C++ 23 - I guess I'd need to drop to at least C++ 20 if I want to move forward.

Problem is, I am using some C++ 23 features. I'll see if I can downgrade somehow.

[andreamancuso]

I commented out a bunch of these:

C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/src/mbgl/util/logging.cpp:74:10: error: no member named 'lock_guard' in namespace 'std'
   74 |     std::lock_guard<std::mutex> lock(mutex);
      |     ~~~~~^
C:/dev/packages/dear-imgui/cpp/deps/maplibre-native/src/mbgl/util/logging.cpp:74:31: error: expected '(' for function-style cast or type construction
   74 |     std::lock_guard<std::mutex> lock(mutex);
      |                     ~~~~~~~~~~^
C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/src/mbgl/util/logging.cpp:74:33: error: use of undeclared identifier 'lock'; did you mean 'Clock'?
   74 |     std::lock_guard<std::mutex> lock(mutex);
      |                                 ^

[andreamancuso]

The good news is, it compiled successfully. Naturally having had to comment out 3 parameters in the draw() method and having commented out mutex locking, I expect all sorts of issues to appear.

[andreamancuso]

I'm afraid all I could manage to accomplish so far is create an instance of MapTilerConfiguration like so:

(printf() emits a message via console.log())

For anything more complex, I'd need help as I'm getting lots of undefined symbol kind of errors. I think I know why, I am not including anything from the platform subfolders.

[andreamancuso]

I'm light years away from where I want to be - but I am celebrating with an IPA. Cheers! 🍻

[andreamancuso]

Another interesting error

In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/src/mbgl/style/expression/value.cpp:3:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/include\mbgl/style/expression/value.hpp:4:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/include\mbgl/style/expression/formatted.hpp:4:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/include\mbgl/style/expression/image.hpp:4:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/include\mbgl/util/color.hpp:3:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/include\mbgl/util/feature.hpp:4:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/deps/maplibre-native/vendor/mapbox-base/include\mapbox/compatibility/value.hpp:3:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/build/vcpkg_installed/wasm32-emscripten/include\mapbox/feature.hpp:3:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/build/vcpkg_installed/wasm32-emscripten/include\mapbox/geometry.hpp:9:
In file included from C:/dev/react-imgui/packages/dear-imgui/cpp/build/vcpkg_installed/wasm32-emscripten/include\mapbox/geometry/geometry.hpp:11:
C:/dev/react-imgui/packages/dear-imgui/cpp/build/vcpkg_installed/wasm32-emscripten/include\mapbox/variant.hpp:332:20: error: no matching function for call to object of type 'mbgl::style::expression::FromMBGLValue'
  332 |             return std::forward<F>(f)(unwrapper<T>::template apply<V>(v));
      |                    ^~~~~~~~~~~~~~~~~~
C:/dev/react-imgui/packages/dear-imgui/cpp/build/vcpkg_installed/wasm32-emscripten/include\mapbox/variant.hpp:336:45: note: in instantiation of function template specialization 'mapbox::util::detail::dispatcher<mbgl::style::expression::Value, std::shared_ptr<std::vector<mapbox::feature::value>>, std::shared_ptr<std::unordered_map<std::string, mapbox::feature::value>>>::apply<const mapbox::feature::value &, mbgl::style::expression::FromMBGLValue>' requested here
  336 |             return dispatcher<R, Types...>::apply(std::forward<V>(v), std::forward<F>(f));

[andreamancuso]

This is working:

target_sources(
    mbgl-core
    PRIVATE
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/gfx/headless_backend.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/gfx/headless_frontend.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/gl/headless_backend.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/i18n/collator.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/i18n/number_format.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/layermanager/layer_manager.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/platform/time.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/asset_file_source.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/database_file_source.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/file_source_manager.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/file_source_request.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/local_file_request.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/local_file_source.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/mbtiles_file_source.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/main_resource_loader.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/offline.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/offline_database.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/offline_download.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/online_file_source.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/sqlite3.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/text/bidi.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/text/local_glyph_rasterizer.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/async_task.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/compression.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/filesystem.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/image.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/jpeg_reader.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/webp_reader.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/logging_stderr.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/monotonic_timer.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/png_reader.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/png_writer.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/run_loop.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/string_stdlib.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/timer.cpp
        ${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/util/utf.cpp
)

This line isn't working:

${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/storage/http_file_source.cpp

due to CURL being unavailable, have to dig deeper

[andreamancuso]

Can't use CURL, will use https://emscripten.org/docs/api_reference/fetch.html

[ntadej]

I propose you define a new platform for WASM.

[andreamancuso]

Makes sense. Apologies for the noise here. I suppose I'm kinda treating this issue almost like a diary. I'll likely do a write-up at some point, summarising findings, plan to action, etc.

[andreamancuso]

I managed to include ${PROJECT_SOURCE_DIR}/platform/linux/src/gl_functions.cpp by enabling -sFULL_ES3 for now. Have yet to assess the performance implications of this.

[ntadej]

I don't mind and it may be useful as a reference for the future 😊

[andreamancuso]

Issues with std::variant encountered on

${PROJECT_SOURCE_DIR}/src/mbgl/style/expression/value.cpp
${PROJECT_SOURCE_DIR}/src/mbgl/util/geojson_impl.cpp

[andreamancuso]

Somehow adding these did not break anything:

${PROJECT_SOURCE_DIR}/platform/glfw/glfw_view.cpp
${PROJECT_SOURCE_DIR}/platform/glfw/glfw_renderer_frontend.cpp
${PROJECT_SOURCE_DIR}/platform/glfw/settings_json.cpp
${PROJECT_SOURCE_DIR}/platform/default/src/mbgl/map/map_snapshotter.cpp

${PROJECT_SOURCE_DIR}/platform/glfw/glfw_gl_backend.cpp

${PROJECT_SOURCE_DIR}/platform/windows/src/headless_backend_egl.cpp
${PROJECT_SOURCE_DIR}/platform/windows/src/gl_functions.cpp

[andreamancuso]

Issues with std::variant encountered on

${PROJECT_SOURCE_DIR}/src/mbgl/style/expression/value.cpp
${PROJECT_SOURCE_DIR}/src/mbgl/util/geojson_impl.cpp

Naturally, due to having commented out these two lines, this fails

[...]
mbgl::HeadlessFrontend frontend({width, height}, static_cast<float>(pixelRatio));

So I need to dig deeper into the issue with std::variant in order to move forward

[ntadej]

std::variant should work fine. At least the OpenGL 2 branch builds fine with Qt.

[andreamancuso]

Understood.

I need a level playing field. I think I'm going to try again using a dummy emscripten project using C++ 17. I need to take deprecations and whatnot out of the equation.

[andreamancuso]

I forked the repo, created a branch and a PR pointing at my fork's main branch for now: https://github.com/andreamancuso/maplibre-native/pull/1

The good news: all those issues with std::variant, etc. are gone. They do appear to be related to having used C++23. This leads to me believe that we could load maplibre native as a side module based on https://emscripten.org/docs/compiling/Dynamic-Linking.html

The bad news: still lots to do but I am so so glad not having to worry about the issues mentioned above.

[andreamancuso]

I already was surpised when you said you compiled libuv for WASM. ;-) Keep us posted!

@louwers Unfortunately you were right 😅

vcpkg truly gave me false hope

[andreamancuso]

Adding these for reference: https://github.com/mapbox/mapbox-gl-js/issues/4835#issuecomment-596499912

https://github.com/emscripten-core/emscripten/issues/10556