Could NOT find ICU trying to build the project from source

[kverrier]

Running into the following issue trying to invoke a modified version of the vvv.sh script:

CMake Error at /usr/local/Cellar/cmake/3.13.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find ICU (missing: ICU_UC_INCLUDE_DIRS ICU_UC_LIBRARIES
  ICU_I18N_INCLUDE_DIRS ICU_I18N_LIBRARIES)
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.13.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindICU.cmake:39 (find_package_handle_standard_args)
  lib/Driver/CMakeLists.txt:40 (find_package)

Here's my modified script:

utils/build-script --release-debuginfo --wasm \
    --llvm-targets-to-build "X86;ARM;AArch64;PowerPC;SystemZ;WebAssembly" \
    --llvm-max-parallel-lto-link-jobs 1 --swift-tools-max-parallel-lto-link-jobs 1 \
    --wasm-wasi-sdk "/Users/kverrier/repos/wasi-sdk" \
    --wasm-icu-uc "todo" \
    --wasm-icu-uc-include "/Users/kverrier/repos/icu4c-wasi/icu/source/common/" \
    --wasm-icu-i18n "todo" \
    --wasm-icu-i18n-include "todo" \
    --wasm-icu-data "todo" \
    --build-swift-static-stdlib \
    "$@"

_note the --wasm-icu-uc-include is pointing to a clone of https://github.com/swiftwasm/icu4c-wasi.

I've been doing a lot of tinkering between failing builds so maybe I need to do a fresh rebuild... Let me know if you have any insights 😄.

[zhuowei]

@kverrier Building the stdlib isn't supported on macOS yet.

For the macOS build:

• I build the toolchain, including wasm stdlib, on a Linux machine
• I build everything except the stdlib on a Mac
• then I copy the stdlib from the Linux build into the Mac build

If you have any ideas for getting the stdlib to build on macOS, we would really appreciate it: @MaxDesiatov was looking into this before and might have some ideas on this as well.

Edit 2: also, the icu path should be pointing to the include dir in ICU: see the Azure Pipelines CI script for info.

[kverrier]

That's quite a build workflow. Thanks for the breakdown.

My hope was getting stuff to build so I could start to help out with the effort 😄. Beyond that, I was curious to try out some of the shiny new Swift 5 features like function builders and property wrappers compiled to WASM. My understanding is the prebuilt swiftc binary from swiftwasm.org doesn't include those yet. I suppose there's nothing stopping me from building the toolchain on mac with the latest stuff but use the slightly older prebuilt linux stdlib.

If you have any ideas for getting the stdlib to build on macOS, we would really appreciate it: @MaxDesiatov was looking into this before and might have some ideas on this as well.

I'll do some tinkering to build the stdlib on the macOS. What was the biggest stumbling block you ran into trying to building stdlib on macOS? Is it the localization thing I shared originally?

P.S. Thanks for the updated ICU directory!

[zhuowei]

@kverrier I highly recommend either setting up a Linux VM or (because I am insane) setup a build computer with Linux, so you can actually try things out without worrying about the build process.

The mac build process is sorta-documented with this script.

[MaxDesiatov]

Hi @kverrier, thank you for trying this out. I agree with @zhuowei, the workflow with compiling the stdlib on Linux works best for now. The main problem is that build-script-impl for both macOS on Linux is complicated and goes through different code paths on these platforms. E.g. ICU is pulled in and built on Linux automatically by the build script, but this doesn't happen on macOS and it attempts to build the system-installed ICU. There's a big old issue about refactoring the build-script-impl on Swift Jira, but there hasn't been any visible progress for years on that front, and refactoring shell scripts is fun 🙃

I'm mostly poking this stuff on Linux these days, building it on AWS and editing/debugging with VS Code Remote SSH extension.

Please do let us know about any issues you stumble upon, we also greatly appreciate any contributions to the project, even small ones, especially around the build system and documentation.

[kverrier]

@zhuowei @MaxDesiatov thanks for the tips! I've spun up a linux VM per your suggestion and using the ci build script for reference to build the project.

I'm running into a new issue though around the Mutex files inside the runtime...

/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:28:2: error: "Implement equivalent of MutexPThread.h/cpp for your platform."
#error "Implement equivalent of MutexPThread.h/cpp for your platform."
 ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:94:3: error: unknown type name 'ConditionHandle'; did you mean 'ConditionVariable'?
  ConditionHandle Handle;
  ^~~~~~~~~~~~~~~
  ConditionVariable
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:69:7: note: 'ConditionVariable' declared here
class ConditionVariable {
      ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:94:19: error: field has incomplete type 'swift::ConditionVariable'
  ConditionHandle Handle;
                  ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:69:7: note: definition of 'swift::ConditionVariable' is not complete until the closing '}'
class ConditionVariable {
      ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:78:25: error: use of undeclared identifier 'ConditionPlatformHelper'
  ConditionVariable() { ConditionPlatformHelper::init(Handle); }
                        ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:79:26: error: use of undeclared identifier 'ConditionPlatformHelper'
  ~ConditionVariable() { ConditionPlatformHelper::destroy(Handle); }
                         ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:85:22: error: use of undeclared identifier 'ConditionPlatformHelper'
  void notifyOne() { ConditionPlatformHelper::notifyOne(Handle); }
                     ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:91:22: error: use of undeclared identifier 'ConditionPlatformHelper'
  void notifyAll() { ConditionPlatformHelper::notifyAll(Handle); }
                     ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:312:3: error: unknown type name 'MutexHandle'
  MutexHandle Handle;
  ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:166:5: error: use of undeclared identifier 'MutexPlatformHelper'
    MutexPlatformHelper::init(Handle, checked);
    ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:168:14: error: use of undeclared identifier 'MutexPlatformHelper'
  ~Mutex() { MutexPlatformHelper::destroy(Handle); }
             ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:179:17: error: use of undeclared identifier 'MutexPlatformHelper'
  void lock() { MutexPlatformHelper::lock(Handle); }
                ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:189:19: error: use of undeclared identifier 'MutexPlatformHelper'
  void unlock() { MutexPlatformHelper::unlock(Handle); }
                  ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:203:28: error: use of undeclared identifier 'MutexPlatformHelper'
  bool try_lock() { return MutexPlatformHelper::try_lock(Handle); }
                           ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:209:5: error: use of undeclared identifier 'ConditionPlatformHelper'
    ConditionPlatformHelper::wait(condition.Handle, Handle);
    ^
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:544:3: error: unknown type name 'ReadWriteLockHandle'; did you mean 'ReadWriteLock'?
  ReadWriteLockHandle Handle;
  ^~~~~~~~~~~~~~~~~~~
  ReadWriteLock
/root/Development/swift-source/swift/include/swift/Runtime/Mutex.h:405:7: note: 'ReadWriteLock' declared here
class ReadWriteLock {
      ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
57 warnings and 20 errors generated.
ninja: build stopped: subcommand failed.
utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

Are there some fake header files missing in my environment?

[zhuowei]

@kverrier I use a hacked-up WASI sdk that pretends to have threads: https://github.com/swiftwasm/wasi-sdk with downloads at https://github.com/swiftwasm/wasi-sdk/releases/tag/20190421.6

[kverrier]

Ah! Since I was copying the linux CI script it's pulling in the debian package but my VM is ubuntu. I should use the linux version of the SDK instead right?

[zhuowei]

@kverrier You can use the .deb package: it's built on Ubuntu 16.04 (which is what Azure CI uses), and it also works on my Ubuntu 18.04 local build machine.

The .tar.gz contains the exact same files; the .deb is just to make it easier to install.

[kverrier]

Interesting... any clue why I'm getting all those compiler errors in the Mutex.h file even when I link in the SDK?

[zhuowei]

@kverrier the errors seems to show that you're using an unmodified WASI sdk, which has threading disabled. Double check that you're using the modified wasi sdk?

[kverrier]

Double check that you're using the modified wasi sdk?

Is there an easy way to check this? I'm definitely passing --wasm-wasi-sdk "/opt/wasi-sdk" (which points to the wasi-sdk you linked earlier) to the build script but maybe it's not piping all the information through?

[zhuowei]

ok, that is weird: try doing a clean build?

[kverrier]

I'll try a clean build 🙃

Would it be insane to put something like:

#error "this is a test"

in the local SDK to make sure that copy is getting pulled in? The only other idea I would have is to debug the log out the CMake variables.

[kverrier]

@zhuowei sad news... Tried to do another clean build and running into the same issue. Any ideas?

[kverrier]

@zhuowei @MaxDesiatov finally figured out this issue 🙃 Turns out my clang checkout was on the swift stable branch instead of stable-swiftwasm. Since my clang checkpoint was wrong, the __wasi__ macro was never defined and that's why the MutexPThread.h was not getting imported.

I made a quick PR just in case it's useful updating the update-checkout script to default to the wasm sheme here: #4.

Now the real work begins since it all builds locally 🤓 Thanks for your assistance!

[zhuowei]

@kverrier Good catch! Thank you so much for the pull request.