[installing boost to webassembly via conan] How to install boost libraries via conan 2 for webassembly

[bruno-j-nicoletti]

What is your question?

Has anyone successfully installed boost via conan 2 for a web assembly target?

I am trying to make my C++ codebase build to web assembly as well as natively. I'm using conan 2 to manage 3rd party libraries, which includes boost. I'm using conan profiles to drive builds for both host or wasm, and using conan install --missing=build ... to do the conan installs.

This works fine for a range of C++ 3rd party libs (eg: catch2), but fails at the first hurdle when conan tries to build boost. It fails on my mac while trying to build bzip2 due to a boost transient dependency. It's basically tries to pass command line options specific to macos to emcc which then falls over.

I'm running emcc 3.1.44, conan 2.0.9, various config files below.

Any help appreciated.

conanfile.py is...

from conan import ConanFile

class BNSConan(ConanFile):
    name = "BNS"
    version = "0.0.1"
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain", "CMakeDeps"

    def requirements(self) :
        self.requires("boost/1.83.0")
        self.requires("catch2/2.13.9")
        self.requires("nlohmann_json/3.5.0")
        self.requires("range-v3/0.12.0")

Webasm conan profile is...

[settings]
build_type=Debug
arch=wasm
compiler=clang
compiler.version=16
compiler.cppstd=23
compiler.libcxx=c++_static
os=Emscripten

[buildenv]
CC=emcc
CXX=em++
LDFLAGS=-s STANDALONE_WASM=1

The errors are

CMake Error at /usr/local/Cellar/cmake/3.25.2/share/cmake/Modules/CMakeTestCCompiler.cmake:70 (message):
  The C compiler

    "/Users/bruno/Work/emsdk/upstream/emscripten/emcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /Users/bruno/Work/kaiju/conanDir/p/b/bzip2c6d7578184ea6/b/build/Debug/CMakeFiles/CMakeScratch/TryCompile-QzmuHM

    Run Build Command(s):/usr/bin/make -f Makefile cmTC_afa0f/fast && /Applications/Xcode.app/Contents/Developer/usr/bin/make  -f CMakeFiles/cmTC_afa0f.dir/build.make CMakeFiles/cmTC_afa0f.dir/build
    Building C object CMakeFiles/cmTC_afa0f.dir/testCCompiler.c.o
    /Users/bruno/Work/emsdk/upstream/emscripten/emcc   -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -mmacosx-version-min=12.6 -fPIE -MD -MT CMakeFiles/cmTC_afa0f.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_afa0f.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_afa0f.dir/testCCompiler.c.o -c /Users/bruno/Work/kaiju/conanDir/p/b/bzip2c6d7578184ea6/b/build/Debug/CMakeFiles/CMakeScratch/TryCompile-QzmuHM/testCCompiler.c
    clang: warning: argument unused during compilation: '-mmacosx-version-min=12.6' [-Wunused-command-line-argument]
    Linking C executable cmTC_afa0f
    /usr/local/Cellar/cmake/3.25.2/bin/cmake -E cmake_link_script CMakeFiles/cmTC_afa0f.dir/link.txt --verbose=1
    /Users/bruno/Work/emsdk/upstream/emscripten/emcc  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -mmacosx-version-min=12.6 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -s STANDALONE_WASM=1  CMakeFiles/cmTC_afa0f.dir/testCCompiler.c.o -o cmTC_afa0f 
    wasm-ld: error: unknown argument: -search_paths_first
    wasm-ld: error: unknown argument: -headerpad_max_install_names
    emcc: error: '/Users/bruno/Work/emsdk/upstream/bin/wasm-ld -o cmTC_afa0f.wasm -search_paths_first -headerpad_max_install_names CMakeFiles/cmTC_afa0f.dir/testCCompiler.c.o -L/Users/bruno/Work/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten /Users/bruno/Work/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/crt1.o -lGL -lal -lhtml5 -lstandalonewasm-nocatch -lstubs-debug -lc-debug -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-debug-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /var/folders/mz/q1551_1d2p9cdkjht_25_h480001m2/T/tmpzwol2amplibemscripten_js_symbols.so --strip-debug --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export-table -z stack-size=65536 --initial-memory=16777216 --max-memory=16777216 --stack-first' failed (returned 1)
    make[2]: *** [cmTC_afa0f] Error 1
    make[1]: *** [cmTC_afa0f/fast] Error 2

[SpaceIm]

I guess your profile is too naive (CMake Toolchain of emsdk is missing among other things).

See how emsdk recipe is written for consumers: https://github.com/conan-io/conan-center-index/blob/230ab447b90a136dca3b2bf45875b39c62136103/recipes/emsdk/all/conanfile.py#L154-L182

Why don't you try to use this recipe in your profile instead?

[settings]
os=Emscripten
arch=wasm
compiler=clang
compiler.version=16
compiler.libcxx=libc++
compiler.cppstd=23
build_type=Release
[tool_requires]
emsdk/3.1.44

By the way c++_static doesn't make sense for emsdk I think, c++_shared & c++_static are specific to Android. AFAIK, the only C++ standard library of emsdk is libc++.

[bruno-j-nicoletti]

Thanks for that!

Adding this to a profile...

[conf]
tools.cmake.cmaketoolchain:toolchain_file={{ os.getenv("EMSDK")+"/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake" }}

Injects the emscripten toolchain into dependent cmakes. My next problem is that the original cmake toolchain file for bzip2 has environment variables needed for the build the package, which aren't in the EMSDK toolchain file. Sigh.

FWIW adding...

[tool_requires]
emsdk/3.1.44

creates a circular reference in the settings and seems to be redundant.

[SpaceIm]

Adding this to a profile...

[conf]
tools.cmake.cmaketoolchain:toolchain_file={{ os.getenv("EMSDK")+"/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake" }}

Injects the emscripten toolchain into dependent cmakes. My next problem is that the original cmake toolchain file for bzip2 has environment variables needed for the build the package, which aren't in the EMSDK toolchain file. Sigh.

You should use tools.cmake.cmaketoolchain:user_toolchain (as pointed out in the link to emsdk recipe in my previous answer) instead of tools.cmake.cmaketoolchain:toolchain_file.

tools.cmake.cmaketoolchain:toolchain_file should never be used honestly because it defeats CMakeToolchain, see the difference:

tools.cmake.cmaketoolchain:toolchain_file: Use other existing file rather than conan_toolchain.cmake one
tools.cmake.cmaketoolchain:user_toolchain: Inject existing user toolchains at the beginning of conan_toolchain.cmake

FWIW adding...

[tool_requires]
emsdk/3.1.44

creates a circular reference in the settings and seems to be redundant.

I've not said that you should mix in your profile your own installation of emsdk with emsdk recipe. You pick one or the other but not both.

[bruno-j-nicoletti]

Thanks again. That fixed the problem with bzip2, but libbacktrace breaks, which is another boost dependent module.

ETA: libbacktrace isn't cmake. Sigh.

[SpaceIm]

Works fine for me with this profile:

[settings]
os=Emscripten
arch=wasm
compiler=clang
compiler.version=16
compiler.libcxx=libc++
compiler.cppstd=23
build_type=Release
[tool_requires]
emsdk/3.1.44

I guess you are still fighting with your own custom profile which still lack important env vars, making your profile fragile with non-cmake build systems.

[bruno-j-nicoletti]

I've just tried with that exact profile and I get...

ERROR: There is a cycle/loop in the graph:
    Initial ancestor: emsdk/3.1.44
    Require: emsdk/3.1.44
    Dependency: emsdk/3.1.44

My conanfile.py looks like...

import os
from conan import ConanFile

class BNSConan(ConanFile):
    name = "BNS"
    version = "0.0.1"
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain", "CMakeDeps"

    def requirements(self) :
        isWebasm = True if self.settings.get_safe("os") == "Emscripten" else False
        if not isWebasm :
            # these wont work in webasm yet
            self.requires("libcoro/0.8")
        self.requires("boost/1.81.0")
        self.requires("catch2/2.13.9")
        self.requires("spdlog/1.11.0")
        self.requires("nlohmann_json/3.5.0")
        self.requires("range-v3/0.12.0")

There's nothing else going in. I'm calling it as...

conan install --output-folder build-wasm . --build=missing -pr:b wasmDebug

The documentation isn't at all clear on what environment variables need to be set!

[SpaceIm]

I don't know why you get this cycle/loop error, maybe because wasmDebug should be your host profile, not your build profile. So it should be -pr:h wasmDebug.

The documentation isn't at all clear on what environment variables need to be set!

I guess conan assumes that users wishing to write custom profiles know what are the common env vars of build systems, like CC, CXX, AR, LD, NM, STRIP, RANLIB etc, but I agree that https://docs.conan.io/2/tutorial/consuming_packages/cross_building_with_conan.html and cross-building examples in current conan v2 documentation could be improved.

[bruno-j-nicoletti]

Doh, I had host and build the wrong way round in my head (I though build meant the system you were building for not building on). Running -pr:h wasmDebug breaks with...

Build requirements
    emsdk/3.1.44#a6d941a26a1c416d25e3505805703a58:9eb5e7905ab73acb05816361e0b394931ce8f7c6 - Invalid

(Thanks ever so much for you help!)

[SpaceIm]

Please share full log saying why configuration is invalid, your conan command, and your profiles (host & build).

[bruno-j-nicoletti]

My default profile is empty, so I'm running this on macos 12.6.7 with Apple clang version 14.0.0 (clang-1400.0.29.202).

Running conan profile show -pr:h wasmDebug I get...

Host profile:
[settings]
arch=wasm
build_type=Debug
compiler=clang
compiler.cppstd=23
compiler.libcxx=libc++
compiler.version=16
os=Emscripten
[tool_requires]
*: emsdk/3.1.44

Build profile:
[settings]

My conanfile.py is now cut down to...

import os
from conan import ConanFile

class BNSConan(ConanFile):
    name = "BNS"
    version = "0.0.1"
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain", "CMakeDeps"

    def requirements(self) :
        self.requires("boost/1.81.0")

If I run conan install --output-folder=build-wasm . --build=missing -pr:h wasmDebug the output is...

======== Input profiles ========
Profile host:
[settings]
arch=wasm
build_type=Debug
compiler=clang
compiler.cppstd=23
compiler.libcxx=libc++
compiler.version=16
os=Emscripten
[tool_requires]
*: emsdk/3.1.44

Profile build:
[settings]

======== Computing dependency graph ========
Graph root
    conanfile.py (BNS/0.0.1): /Users/bruno/Work/kaiju.blah/conanfile.py
Requirements
    boost/1.81.0#4d2d9301e9b9113e959c41295f076c52 - Cache
    bzip2/1.0.8#411fc05e80d47a89045edc1ee6f23c1d - Cache
    libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c - Cache
    zlib/1.2.13#e377bee636333ae348d51ca90874e353 - Cache
Build requirements
    b2/4.9.6#53783bdbbdbb6c13732b25f1f6ea6c58 - Cache
    emsdk/3.1.44#a6d941a26a1c416d25e3505805703a58 - Cache
    nodejs/16.3.0#ecc03e2039d93ec8c6df784f8533c8a2 - Cache

======== Computing necessary packages ========
boost/1.81.0: Checking 9 compatible configurations:
boost/1.81.0: '82853510ed265503ef84456cad531e16d7a310f7': compiler.cppstd=11
boost/1.81.0: '96771ddc32e8ade49dec6003a72c76cb4fd12bc9': compiler.cppstd=gnu11
boost/1.81.0: 'ae97999298728dcab916aa0729b2cb525d78de44': compiler.cppstd=14
boost/1.81.0: '2b8dd59136ff5a40d167e4bbff5198322a373381': compiler.cppstd=gnu14
boost/1.81.0: '0ba00b1a7b93f7b56268173bf4a8abe9cb823f72': compiler.cppstd=17
boost/1.81.0: '377c7811acb0fdbdc83c16bcb89a633d05f8b58f': compiler.cppstd=gnu17
boost/1.81.0: '3c8942c020390bbdf88777078f392776c7cd3a93': compiler.cppstd=20
boost/1.81.0: '90c247e2c77234e08a5580c447bb1e62be3a494e': compiler.cppstd=gnu20
boost/1.81.0: '0459021e0113417a4d53f7551cf2e0ce8d2bcb80': compiler.cppstd=gnu23
Requirements
    boost/1.81.0#4d2d9301e9b9113e959c41295f076c52:1bc091b14e4b6ef67a7dd341fb3502f5efdcb590 - Build
    bzip2/1.0.8#411fc05e80d47a89045edc1ee6f23c1d:6d96f99e9965a4797bd36a3301a7da9699f8681c - Build
    libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:e3fac8d3f9347da32209118b429e89851f8a417d - Build
    zlib/1.2.13#e377bee636333ae348d51ca90874e353:e3fac8d3f9347da32209118b429e89851f8a417d - Build
Build requirements
    b2/4.9.6#53783bdbbdbb6c13732b25f1f6ea6c58:da39a3ee5e6b4b0d3255bfef95601890afd80709 - Invalid
    emsdk/3.1.44#a6d941a26a1c416d25e3505805703a58:a9a658c22a013f191ff66fbfd9e3ba321eb98199 - Invalid
Skipped binaries
    nodejs/16.3.0

======== Installing packages ========
ERROR: There are invalid packages:
b2/4.9.6: Invalid: 'settings.os' value not defined
emsdk/3.1.44: Invalid: 'settings.os' value not defined
(venv3_11) 142 bruno@cheesoid:[main]:~/Work/kaiju.blah ? 

[SpaceIm]

My default profile is empty

That's the problem. Create a default profile (or manually pass a build profile), otherwise it can't work.

[settings]
os=Macos
arch=x86_64
compiler=apple-clang
compiler.version=14
compiler.libcxx=libc++
compiler.cppstd=17
build_type=Release
[options]
[tool_requires]
[conf]

You can also call conan profile detect --force if you wish conan to autodetect your default profile. It's explained in the first tutorial of conan: https://docs.conan.io/2/tutorial/consuming_packages/build_simple_cmake_project.html

[bruno-j-nicoletti]

Thank you. That seems to have fixed it. Currently running. I don't build x86 on macos, only linux, so I didn't bother with a macos profile. It's currently pulling down node (!). I'll see how it goes.

[bruno-j-nicoletti]

That got alot further but now fails in boost jam (shudder, why are they still using jam?).

Error: ambiguity found when searching for best transformation
Trying to produce type 'SEARCHED_LIB' from: 
Generators that succeeded:
 -  searched-lib-generator
 -  emscripten.searched-lib-generator
First generator produced: 
 -  { %.no-action-z.SEARCHED_LIB }
Second generator produced: 
 -  { %.no-action-z.SEARCHED_LIB }

I'm giving up on building boost, the two libraries I'm currently using I can work around with a bit less pain than this.

Thank you very much for you help, as it should fix a few issues I've had with a couple of other libraries.

[SpaceIm]

That got alot further but now fails in boost jam (shudder, why are they still using jam?).

Error: ambiguity found when searching for best transformation
Trying to produce type 'SEARCHED_LIB' from: 
Generators that succeeded:
 -  searched-lib-generator
 -  emscripten.searched-lib-generator
First generator produced: 
 -  { %.no-action-z.SEARCHED_LIB }
Second generator produced: 
 -  { %.no-action-z.SEARCHED_LIB }

Yes, I can reproduce this issue. There might be a hack we could add in boost recipe: https://stackoverflow.com/a/47751199/8390473 (in comments of this answer), or the one of https://lists.boost.org/boost-users/2018/07/88788.php by using gcc toolset instead of emscripten toolset for b2.

See also https://lists.boost.org/boost-users/2018/07/88788.php. It seems to be a bug of emscripten toolset in b2?

/cc @TimSimpson @anagno @ovostrikov who have contributed to emsdk build fixes in boost recipe in the past.

[ovostrikov]

That got alot further but now fails in boost jam (shudder, why are they still using jam?).

Error: ambiguity found when searching for best transformation
Trying to produce type 'SEARCHED_LIB' from: 
Generators that succeeded:
 -  searched-lib-generator
 -  emscripten.searched-lib-generator
First generator produced: 
 -  { %.no-action-z.SEARCHED_LIB }
Second generator produced: 
 -  { %.no-action-z.SEARCHED_LIB }

Yes, I can reproduce this issue. There might be a hack we could add in boost recipe: https://stackoverflow.com/a/47751199/8390473 (in comments of this answer), or the one of https://lists.boost.org/boost-users/2018/07/88788.php by using gcc toolset instead of emscripten toolset for b2.

See also https://lists.boost.org/boost-users/2018/07/88788.php. It seems to be a bug of emscripten toolset in b2?

/cc @TimSimpson @anagno @ovostrikov who have contributed to emsdk build fixes in boost recipe in the past.

I've encountered same issue previously and patching /src/tools/emscripten.jam in b2 sources according to this comment https://github.com/boostorg/regex/issues/59#issuecomment-828707747 fixed the problem.

[bruno-j-nicoletti]

That's fixed it. I just needed to patch the emscripten.jam file and disable the stacktrace boost lib and it is works.

Phew. Thanks for all the help.

[bruno-j-nicoletti]

Last stupid question. I patched the emscripten.jam file by hand to get it to work. As a conan consumer of boost and emscripten, I cannot figure out how I script in the patching of that file during a conan install. You can patch files if you are a producer of a conan package in the ConanFile.source method, but as a consumer that isn't run.

[SpaceIm]

Indeed, the idea would be to submit a PR in conancenter to dynamically patch this file (at some point in build() method I guess), so that it can benefit to all consumers.

[bruno-j-nicoletti]

Given that the emscripten.jam file has significantly changed in the b2 github repo, and figuring out how to reproduce the whole toolchain with the mainline version of b2 to reproduce the failure on main would be difficult, I've raise in an issue as opposed to a PR, but outlining the fixes that worked and and the whole method to repro the issue.