Closed geelen closed 4 years ago
Any news? (:
Does this help anything? https://hacks.mozilla.org/2016/10/webassembly-browser-preview/
@kripken @ephemer I'd love to push this forward if you could point me in the right direction!
@paulofaria thanks for your enthusiasm. There's a fair bit of work being done on the swift build system at the moment to allow for cross compilation, that will bring us closer to building for webassembly.
It's a lot of work that involves changing a running system at its core but it'll get there
any update?
@goloveychuk not that I'm aware of
Any news ?
I don't know how relevant this is for the question, but Swift Package Manager now supports cross compilation. I've built X toolchains for compiling Raspberry Pi Swift binaries on macOS and the reverse. Both however require the availability of Swift binaries (which we build on Raspi instead of x-compiling Swift itself).
My understanding is that CLang itself is always a cross compiler and comes w/ all supported targets builtin (i.e. the standard macOS clang can build for the Raspi armhf target out of the box, you just have to bring the toolchain). So if
Upstream LLVM has a WebAssembly triple now.
I suppose a lot may come for free. Has that 'upstream' be part of a release yet? Maybe the Xcode 9 one includes it? (is there a way to list the supported architectures by clang?)
Also, I think Swift 4 SPM defaults to building static libraries now.
Swift running in the browser at native-code speed? Heck yeah, let's make this happen!
I've downloaded the swift compiler and the emscripten sdk. I've successfully used emscripten to build and run "Hello world" in C. I tried passing "-target wasm32-wasm32-wasm32" to the latest Xcode 9 swiftc but it didn't work.
I'd like to do the process that @ephemer suggested above but I'll need a little guidance. Can you elaborate a bit on what the steps would be?
Oh hey, wasm in WebKit: https://webkit.org/blog/7691/webassembly/
Also, the Xcode 9 build of Clang supports the wasm32 and wasm64 targets:
% xcrun clang --version
Apple LLVM version 9.0.0 (clang-900.0.26)
Target: x86_64-apple-darwin16.6.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
% xcrun clang -emit-llvm --target=wasm32 -Oz fib.c -c -o fib.bc
...any updates?
I tried to compile swift llvm ir code to wasm via emscripten - some syntax problems. I guess we should try to build apple/swift-llvm and upstream llvm from source and try to compile ir code. But this will likelly fail, sine apple frontend itself should support wasm32 target. This will require porting apple/swift for wasm32 target (same it was with android, but looks like, will requiremore significant changes)
Achieved small success.
1) build llvm from sources. https://github.com/yurydelendik/wasmception
2) swiftc -emit-ir -Onone code.swift -o code.ll
3) open code.ll
and remove
!5 = !{i32 6, !"Linker Options", !6}
!6 = !{!7, !8, !9}
!7 = !{!"-lswiftCore"}
!8 = !{!"-lswiftSwiftOnoneSupport"}
!9 = !{!"-lobjc"}
and !5
from
!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !10, !11}
(looks like swift-llvm is older than upstream llvm)
4) try llc -mtriple wasm32 code.ll
you will see WebAssembly doesn't support non-C calling conventions
So problem is that swift using swiftcc
calling convention. Wasm supports only ccc
calling convention.
https://llvm.org/docs/LangRef.html#calling-conventions
(if you comment this function llc
will produce .s
file.)
@kripken do you know, is it really possible to bring swiftcc
convention to wasm target?
The calling conventions are mostly for the backends, so it should be possible to just ignore it. That is, if swift didn't emit it, it should be ok, and for now you can strip it from the IR manually for testing. (There are exceptions like tail call optimizations that are done on IR and depend on the convention, but I don't think that would be an issue here.)
If you do that, is the output runnable? The simplest way to check is probably to use the wasm backend with emscripten instead of calling llc and s2wasm etc. yourself, that is, run emcc fixed.ll
for LLVM IR with the calling convention stuff fixed up, and where emcc is set up to use latest LLVM built with the wasm backend (can set the LLVM
env var to point it to that build, or edit ~/.emscripten
).
ok, i tried emcc. After swiftc we have
define i32 @main(i32, i8**) #0 {
entry:
%2 = bitcast i8** %1 to i8*
ret i32 0
}
define hidden swiftcc void @_T04code4mainyyF() #0 {
entry:
br label %0
; <label>:0: ; preds = %entry
br label %2
; <labe
...
So we have signature and implementation separately. Emcc catches first function, so we have no output. Ok, I merged it to one function. This is whole code.
; ModuleID = 'code.ll'
source_filename = "code.ll"
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "asmjs-unknown-emscripten"
%struct._SwiftEmptyArrayStorage = type { %struct.HeapObject, %struct._SwiftArrayBodyStorage }
%struct.HeapObject = type { %struct.HeapMetadata*, %struct.InlineRefCounts }
%struct.HeapMetadata = type opaque
%struct.InlineRefCounts = type { i32, i32 }
%struct._SwiftArrayBodyStorage = type { i64, i64 }
%swift.type = type { i64 }
%swift.bridge = type opaque
%Any = type { [24 x i8], %swift.type* }
%TSi = type <{ i64 }>
@_swiftEmptyArrayStorage = external global %struct._SwiftEmptyArrayStorage, align 8
@_T0SiN = external global %swift.type, align 8
@__swift_reflection_version = linkonce_odr hidden constant i16 3
@llvm.used = appending global [1 x i8*] [i8* bitcast (i16* @__swift_reflection_version to i8*)], section "llvm.metadata", align 8
define i32 @main(i32, i8**) #0 {
entry:
%2 = bitcast i8** %1 to i8*
; ret i32 0
; br label %0
; <label>:0: ; preds = %entry
; br label %2
; <label>:1: ; preds = %2
br label %3
; <label>:2: ; preds = %0
; br label %1
; <label>:3: ; preds = %1
%4 = call swiftcc { %swift.bridge*, i8* } @_T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5(i64 1)
%5 = extractvalue { %swift.bridge*, i8* } %4, 0
%6 = extractvalue { %swift.bridge*, i8* } %4, 1
%7 = call %swift.bridge* @swift_bridgeObjectRetain(%swift.bridge* %5) #2
call void @swift_bridgeObjectRelease(%swift.bridge* %5) #2
%8 = bitcast i8* %6 to %Any*
%9 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 1
store %swift.type* @_T0SiN, %swift.type** %9, align 8
%10 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 0
%11 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 0
%12 = bitcast [24 x i8]* %11 to %TSi*
%._value = getelementptr inbounds %TSi, %TSi* %12, i32 0, i32 0
store i64 578, i64* %._value, align 8
%13 = call swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA0_()
%14 = extractvalue { i64, i64, i64 } %13, 0
%15 = extractvalue { i64, i64, i64 } %13, 1
%16 = extractvalue { i64, i64, i64 } %13, 2
%17 = call swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA1_()
%18 = extractvalue { i64, i64, i64 } %17, 0
%19 = extractvalue { i64, i64, i64 } %17, 1
%20 = extractvalue { i64, i64, i64 } %17, 2
call swiftcc void @_T0s5printySayypGd_SS9separatorSS10terminatortF(%swift.bridge* %5, i64 %14, i64 %15, i64 %16, i64 %18, i64 %19, i64 %20)
ret i32 0
}
; Function Attrs: noinline
declare swiftcc void @_T0s5printySayypGd_SS9separatorSS10terminatortF(%swift.bridge*, i64, i64, i64, i64, i64, i64) #1
declare swiftcc { %swift.bridge*, i8* } @_T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5(i64) #0
declare %swift.bridge* @swift_bridgeObjectRetain(%swift.bridge*)
declare void @swift_bridgeObjectRelease(%swift.bridge*)
; Function Attrs: noinline
declare swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA0_() #1
; Function Attrs: noinline
declare swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA1_() #1
attributes #0 = { "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "target-cpu"="core2" "target-features"="+cx16,+fxsr,+mmx,+sse,+sse2,+sse3,+ssse3,+x87" }
attributes #1 = { noinline "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "target-cpu"="core2" "target-features"="+cx16,+fxsr,+mmx,+sse,+sse2,+sse3,+ssse3,+x87" }
attributes #2 = { nounwind }
!llvm.module.flags = !{!0, !1, !2, !3, !4, !10, !11}
!0 = !{i32 1, !"Objective-C Version", i32 2}
!1 = !{i32 1, !"Objective-C Image Info Version", i32 0}
!2 = !{i32 1, !"Objective-C Image Info Section", !"__DATA, __objc_imageinfo, regular, no_dead_strip"}
!3 = !{i32 4, !"Objective-C Garbage Collection", i32 1280}
!4 = !{i32 1, !"Objective-C Class Properties", i32 64}
!10 = !{i32 1, !"PIC Level", i32 2}
!11 = !{i32 1, !"Swift Version", i32 5}
It compiles with emcc but writes messages like
warning: unresolved symbol: _T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5
So, problem with linking. This could be caused by swiftcc
or because I removed
!5 = !{i32 6, !"Linker Options", !6}
!6 = !{!7, !8}
!7 = !{!"-lswiftCore"}
!8 = !{!"-lswiftSwiftOnoneSupport"}
Ok, yeah, that's a runtime linking issue then (the !5 = !
etc. things at the end are metadata, and can be ignored for now). Swift has a runtime that implements necessary things, like I guess _T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5
allocates an uninitialized array, and there is an "Objective C bridge" of some sort.
Is there an open source version of the swift runtime library? If so then the next step is to build that (to bitcode, and link that to the main program). That runtime might need to be ported to wasm as well, but if it's already portable enough to run on linux and macos then it might just work, or might only take a little work.
there is an "Objective C bridge" of some sort.
It may be better to not use the macOS compiler but do this on Linux. On macOS Swift has Objective-C integration, which is not there on Linux.
Is there an open source version of the swift runtime library?
At least the Linux stuff is all open source and available here: https://github.com/apple/swift
ok, let's forget for now about linking, calling conventions, std/runtime libs. Here is simple swift code without using stdlib and cleaned llvm ir. https://gist.github.com/goloveychuk/76f842045f28a19c939b210937dce2d4 It fails both with upstream llvm and built-in llvm
Built-in:
Not handled: i64 sub (i64 ptrtoint ([13 x i8]* @2 to i64), i64 ptrtoint (<{ i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 }>* @_T04code5StrctVMn to i64))
LLVM ERROR: FlattenGlobals: ConstantExpr opcode not handled: sub
Which is ok, since emscripten's mostyle targeting c/cpp code.
With upstream llvm:
DEBUG:root:emscript: binaryen s2wasm: /Users/badim/github/wasm/emsdk-portable/clang/e1.37.33_64bit/binaryen/bin/s2wasm /tmp/emscripten_temp/tmp0FUxXQ.wb.s --emscripten-glue --global-base=1024 --initial-memory=16777216 -l /Users/badim/.emscripten_cache/wasm/wasm_libc_rt.a -l /Users/badim/.emscripten_cache/wasm/wasm_compiler_rt.a --trap-mode=allow
[[object_alias:]]:
==========
.int32 2
.int32 3
.int32 (.L__unnamed_2-_T04code5StrctVMn)
==========
Included .s
in gist
Looks like that's a bug in s2wasm. It's actually being deprecated, though, so it's probably best to look at the new path, which is to use lld. Try building with EMCC_EXPERIMENTAL_USE_LLD=1
in the environment.
Great! It compiled with several warnings
warning: unresolved symbol: _T0SiN
warning: unresolved symbol: _swift_slowAlloc
warning: unresolved symbol: _swift_slowDealloc
caused by linking problem. It running and fails in runtime bacause can't find those functions. Next step - fix linking with stdlib/runtime.
Great progress!
I have been doing trying to get everything to compile and linking.
Here is what I have so far (without success) https://github.com/patcheng/swift/tree/swift-to-wasm
But it requires https://github.com/patcheng/clang/tree/swift-to-wasm and https://github.com/patcheng/llvm/tree/swift-to-wasm
I had to build my own compile-rt, libcxx, libcxxabi and icu4c.
Here are the things I found (and attempted to address in my fork):
Things are "compiling" and "linking". Known issues:
@patcheng nice work - sounds promising. It seems that swift-clang
& swift-llvm
are now up to date with release_60
which includes WASM changes. You should be able to have your branches forked from those instead.
I've rebased your changes on those here if you want to use it:
Note, I did have to resolve some conflicts, so you may want to verify. Also I didn't have what I needed to resolve conflicts in combine-sdiv.ll
so you may need to resolve that yourself (I suspect it's mostly generated code).
It'd be great to get some of these changes merged onto https://github.com/apple 😄
What is the status on this? Is there any way in which I can help? If so, where should I start?
LLVM 7.0 was released yesterday with a lot of progress for WebAssembly target support. While the backend is still marked as experimental, not as much patching of upstream LLVM repositories should be needed. I've pulled 7.0 updates into Apple swift's fork repositories and rebased the changes in the main swift
repository by @patcheng, while also fixing a few compilation issues.
Now the whole fresh toolchain with patches from above comments compiles for me, although I haven't checked yet if the actual WebAssembly support has improved. Going to look into that soon. I'm doing the work in these forks and branches:
https://github.com/maxdesiatov/swift https://github.com/maxdesiatov/swift-compiler-rt https://github.com/maxdesiatov/swift-llvm https://github.com/maxdesiatov/swift-clang https://github.com/maxdesiatov/lld
Hey @maxdesiatov, would love to hear of any updates.
Hey @paulshapiro, I appreciate your interest! I've got most if not all of Swift standard library compiling, as well as patches from @patcheng. I've also added support for Swift calling conventions when targeting WebAssembly in my clang and llvm forks. The main problem now is linking the standard library with its dependencies when compiled to WebAssembly. The build process gets a bit complicated though, as Swift stdlib also depends on libc (musl fork supporting WebAssembly in this case), libc++ and libc++abi and ICU all compiled and linked together for WebAssembly target from scratch. It also looks like all of this can only be linked with lld.
Unfortunately, a build system used within Swift compiler infrastructure is not very modular, it uses a mix of shell scripts, Python scripts and CMake. In addition to that, it doesn't look like cross-compiling the whole toolchain from scratch is supported well.
I'm currently getting changes to the build system polished so that wider audience could just clone my fork and run a reproducible build. This doesn't mean that Swift standard library will be compiled, linked and working for WebAssembly end-to-end at that point, but at least the build system issues would become reproducible and allow more people to contribute their fixes to these problems.
TWIMC, even when we're able to compile a few simple Swift executables to WebAssembly, there still a ton of infrastructure work than needs to be done for all this to work smoothly: Swift Package Manager support, helpers for managing multiple cross-compilation toolchains, IDE support, better bridging to JavaScript etc.
In my opinion, one of the most pressing issues is that it's not easy to produce a self-contained statically linked binary written in Swift be it a WebAssembly binary or an ELF binary for Linux. This is going to be a quite serious problem until we have dynamic linking working well in WebAssembly, which is probably not going to happen soon.
If you'd like to be able to easily distribute a self-contained WebAssembly builds of your Swift code, please support work being done on this (in the increasing order of complexity):
build-script-impl
that controls the build process of Swift compiler infrastructureThanks!
I think I finally got the build script modified so that musl
, icu4c
, libcxx
and libcxxabi
are pulled and built automatically with a few more cross-compilation flags set. Unfortunately, there are still build errors caused by missing directories, which need to be investigated.
At least, no more manual steps or additional custom scripts are required now. You can reproduce this WebAssembly build (and errors until those are fixed) with this short script:
mkdir swift-source && cd swift-source && \
git clone https://github.com/maxdesiatov/swift.git && \
./swift/utils/update-checkout --clone --scheme wasm-next \
--target-platform WebAssembly && \
time ./swift/utils/build-script --release-debuginfo --skip-build-benchmarks \
--webassembly --libicu
This will pull the correct forks with correct branch scheme configured.
To rerun the build after you've done any changes you can do this:
time ./swift/utils/build-script --release-debuginfo --skip-build-benchmarks \
--webassembly --libicu
This will also print the total build time, not the check out time though, but check out is supposed to be done only once anyway.
Be prepared for long build times for cold builds: on 2018 15" MBP with top CPU and RAM config it takes around 40 minutes until failing. I anticipate the complete successful build to take much more than this as it needs to assemble both the host macOS and the cross-compiled WebAssembly toolchains.
Thank you all for following. I'll post updates here when I get any significant fixes pushed to my fork. PRs welcome!
For future reference, this is the error message during Swift stdlib compilation that's currently blocking me:
FAILED: stdlib/public/core/webassembly/wasm32/Swift.o
cd /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/bin/python /Users/maxd/Documents/swift-source-wasm/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt -- /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -c -sdk -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source-wasm/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library -o /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt
<unknown>:0: error: no such file or directory: 'wasm32-unknown-unknown-wasm'
[1039/1331] Generating /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule
FAILED: lib/swift/webassembly/wasm32/Swift.swiftmodule lib/swift/webassembly/wasm32/Swift.swiftdoc lib/swift/webassembly/wasm32/Swift.swiftinterface
cd /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftdoc && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftinterface && /usr/bin/python /Users/maxd/Documents/swift-source-wasm/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt -- /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -emit-module -o /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule -emit-parseable-module-interface -sdk -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source-wasm/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt
<unknown>:0: error: no such file or directory: 'wasm32-unknown-unknown-wasm'
[1049/1331] Compiling /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/macosx/x86_64/Swift.o
ninja: build stopped: subcommand failed.
./swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
error: no such file or directory: 'wasm32-unknown-unknown-wasm'
my ignorant suggestion:
-target=wasm32-unknown-unknown-wasm
or
-target:wasm32-unknown-unknown-wasm
instead of
-target wasm32-unknown-unknown-wasm
?
It doesn't look like the argument is passed incorrectly, otherwise I'd expect line-directive
or its proxied swiftc
invocation to explicitly complain about the incorrect argument.
My current guess is that something's missing in build-script-impl
or one of the numerous CMake files and their cross-compilation settings. Those seem to fail to create this directory within the main build directory tree. I plan to have a closer look at this line-directive
script and possibly to debug it and swiftc
to see where exactly the attempt to address that directory occurs and what's the absolute path of it. Then we'd need to retrace the build scripts to understand what's the best setting to tweak.
nice progress!
according to swiftc -h
:
-sdk <sdk> Compile against <sdk>
so, looks like <sdk>
is missing.
looking at SwiftConfigureSDK.cmake, configure_sdk_windows
defines:
set(SWIFT_SDK_${prefix}_ARCH_${arch}_PATH "/")
but we are not setting the variable in configure_sdk_webassembly
.
Thank you @patcheng, that's a great shout!
I've fixed that missing directory problem and a few more header import errors following after that in my fork. Here's the error I'm currently stuck at:
[1049/1331] Compiling /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o
FAILED: stdlib/public/core/webassembly/wasm32/Swift.o
cd /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/bin/python /Users/maxd/Documents/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/ce5Xt.txt -- /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -c -sdk / -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library -o /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o @/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/ce5Xt.txt
<unknown>:0: error: file '/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm' is not a valid precompiled module file
/Users/maxd/Documents/swift-source/swift/stdlib/public/core/ArrayBody.swift:18:8: error: no such module 'SwiftShims'
import SwiftShims
Looking at this precompiled module file we can see that it's compiled to WebAssembly:
% file module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm
module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm: WebAssembly (wasm) binary module version 0x1 (MVP)
Not sure if that's any good, a bit more googling leads to this clang doc on precompiled modules. There's also a mention of an unmerged patch for WebAssembly precompiled modules here and an unresolved clang bugreport probably related to this problem, both posted by @patcheng by the way 👍
Probably need to try and apply that patch or figure another way to make clang consume these precompiled WebAssembly modules. Or maybe these modules shouldn't be compiled to WebAssembly altogether? 🤔
Hey @MaxDesiatov, a big thanks for all the work you've been doing on this!
I was wondering what the current status of compiling Swift to WASM is right now. I lack much experience in this area, but I'm very excited to potentially see support for this, so I would be happy to attempt contributing toward these efforts, given some guidance on what needs to be worked on.
Thanks!
I’m pretty sure the precompiled modules should be readable to the cross-compile host (in your case Mac OS?) so shouldn’t be in web assembly format.
This is a common pain when trying to cross-compile generally (host vs target), and the last I looked the swift toolchain was doing weird things in this regard because it doesn’t follow CMake’s standard- Apple was having toolchain build performance issues internally using it in that way.
So to fix your issue it’s possible it’s an LLVM bug but it may also be an issue with the CMake scripts (or what is being passed to them by build-script-impl). I haven’t looked at the build system in a while.
I’m still tied up with some other projects but I do look here periodically and will try to share what limited experience I have with the build system.
I just discovered that @ddunbar has been creating PRs to add WASM support to Swift:
https://github.com/apple/swift-clang/pull/235 https://github.com/apple/swift/pull/20684 https://github.com/apple/swift/pull/20687
Just a few, very minimal initial ones... this bug has the status: https://bugs.swift.org/browse/SR-9307
Hi all, sorry for the delayed reply. There are few more blocking things that I've discovered while trying to make stdlib and a few basic examples compile:
The standard library is built under assumption of availability of multi-threading and atomics, which is not true for WASM and won't be for long time until all browsers support multi-threaded WASM (and multi-threaded WASM isn't even standardized yet AFAIR?). As was mentioned in the linked Swift Forums topic, the best option is probably to create stubs/shims for all files that do #include <atomic>
or maybe find some libc++ implementation that provides those stubs for us?
The LLVM code that Swift's compiler generates in IRGen
also contains instructions for atomics. Those need to be omitted during code generation as they can't be lowered to WASM binary code due to the same reasons mentioned in 1.
Some of the instructions generated by Swift's IRGen
also specify memory offsets that can't be lowered to WASM. I didn't have enough time to figure out if that's a temporary limitation of LLVM WASM backend or has to be fixed in Swift's IRGen
similar to 2.
Hope this helps.
It looks like short-term it would be great if we could push apple/swift#20684 PR towards being merged. Reviewing its code, commenting on the review, upvoting the PR, pinging the authors/reviewers once a week or at least once a month and sharing to your friends so that they do the same would definitely help. And yes, the Apple's contribution guide for the Swift compiler says we can ping people once a week 😃
- Ping the change. If it is urgent, provide reasons why it is important to get this change landed and ping it every couple of days. If it is not urgent, the common courtesy ping rate is one week. Remember that you’re asking for valuable time from other professional developers.
Here's also quick guide if anyone's interested in investigating the rest of the issues I mentioned above. The gist is that you need to compile a Swift toolchain that supports -target wasm32-unknown-unknown-wasm
flag passed to the Swift compiler (which is what apple/swift#20684 is about). It's available in my wasm-next
branch here https://github.com/MaxDesiatov/swift/tree/wasm-next, you can clone the dependencies after checking out that main forked repository this way: ./swift/utils/update-checkout --clone --scheme wasm-next
.
Or you can try @ddunbar's branch from the PR, but you need to check if it pulls fresh LLVM/Clang when doing update-checkout
from that branch.
I haven't compiled any of these branches for quite some time so that's why some rebasing on top of the latest code and resolving conflicts might be needed.
After you've got the toolchain compiling successfully you can try compiling MicroStdlib test:
% ./build/Ninja-DebugAssert+stdlib-ReleaseAssert/swift-macosx-x86_64/bin/swiftc \
-target wasm32-unknown-unknown-wasm -c -force-single-frontend-invocation \
-parse-as-library -parse-stdlib -module-name Swift -emit-module \
-emit-module-path Swift.swiftmodule -o Swift.ll \
swift/validation-test/stdlib/MicroStdlib/Inputs/Swift.swift
This will generate a Swift.ll
file with LLVM instructions that we need to "lower" to WebAssembly binaries. You can even start with commenting out all of the code in swift/validation-test/stdlib/MicroStdlib/Inputs/Swift.swift
and compiling an empty file. If that works, you can uncomment the most simple code line by line to proceed towards next issue.
If all goes well at the previous stage you can then compile Swift.ll
file to WebAssembly with ./build/Ninja-DebugAssert+stdlib-ReleaseAssert/llvm-macosx-x86_64/bin/llc Swift.ll
. That's where you might stumble upon problems with instructions not supported by the WebAssembly LLVM backend. It looks like most of the time it's Swift compiler generating instructions that don't make sense for WASM (atomics and probably address offsets, but the latter needs to be confirmed). You can also refer to LLVM guide for more details on these instructions.
Instructions that don't make sense should be fixed in IRGen
code, otherwise we need to report problems with WebAssembly backend to upstream LLVM, but fixing those locally in forked LLVM repositories in the meantime is fine too. The earlier those problems are reported, fixed and upstreamed to the main LLVM repository (not Apple's), the higher is the chance they end up in the imminent LLVM 8.0 release and we won't need to maintain fixes in our forks anymore.
- The standard library is built under assumption of availability of multi-threading and atomics, which is not true for WASM and won't be for long time until all browsers support multi-threaded WASM (and multi-threaded WASM isn't even standardized yet AFAIR?). As was mentioned in the linked Swift Forums topic, the best option is probably to create stubs/shims for all files that do
#include <atomic>
or maybe find some libc++ implementation that provides those stubs for us?
Maybe could be simulated using green threads ?
Maybe could be simulated using green threads ?
It's not that the standard library needs green threads or multi-threading to make it work, it's just that reference counting and the rest of the runtime were never written for platforms like WASM (or AVR for example) in mind, so it guards everything with atomics and synchronization assuming you always have to be safe when passing ref-counted objects around. On WASM that's not a problem as long as you don't have threads, we could even throw out all of the atomics and synchronization, the problem would be to keep that forked version maintained. That's why I'd prefer to have the shims compiled conditionally with upstream stdlib source code if/when we have shims available for atomics. And this is what people from the Swift core team recommended to do anyway:
It would be interesting to add a configuration flag to disable threading support in the "normal" runtime and standard library. Nobody's done that yet to my knowledge. For that specific issue, though, it is incorrect to avoid using
in a single-threaded system; I'm not sure why that #if is there. is still meaningful for a single-threaded system for atomicity relative to signals and other preemption mechanisms. It’s unfortunate libc++ doesn’t have a single-threaded atomic implementation, but having a stubbed-out version of atomic, mutex, and other synchronization primitives would be nice to minimize the amount of conditional code we need in the runtime itself.
I left a comment on the forum, gist being that wasm has plans for atomics, but not any time soon, so stubbing <atomic>
sounds like the most pragmatic choice.
I’d avoiding trying to fork the things to not use atomics in the stdlib/runtime because it’s much more likely to change than atomics, and you can write/maintain NOP functions much more easily than maintaining a fork of the standard library.
Thanks for all the hard work trying to get this to work - Swift is a fantastic language that I would love to see running in the browser (although the 5 megabyte bundle size will make me think twice).
I spent the past week looking into this: it looks like LLVM/WebAssembly's support for custom data sections isn't mature yet, so Swift's runtime type reflection won't work.
I got to the same stage as @patcheng's and @MaxDesiatov's ports: specifically, I can compile a simple Swift function that doesn't use the stdlib and run it in Firefox.
but I'm stuck here now as well.
I had to comment out a lot of stuff in LLVM to get asserts to stop firing when writing the type information sections to the wasm file.This results in invalid wasm files when I compile any Swift file with a class or struct declaration in it... including the stdlib.
While it's possible to disable runtime type information in Swift (with the -disable-reflection-metadata flag), this would break pretty much everything in the stdlib. (I remember from the Swift 2 era that even println("hello world")
would crash without runtime type info)
I know @patcheng sent some patches to LLVM to try to resolve this, but I'm not sure what they do or whether more changes are needed.
I wonder how other language with runtime types support, such as Go or Rust's Any trait, encode their type information in WebAssembly.
I wonder if using something like Binaryen instead of LLVM would allow us to write the metadata properly 🤔
I know that Binaryen IR doesn't even use SSA, this would probably require writing whole codegen pass for it, but maybe still worth it? Does anyone here have any experience with Binaryen and can provide their opinion on this?
Hi zhuowei,
I work on the llvm backend for WebAssembly. We are certainly interested in any bugs you have found related to custom data sections. Could you open a bug https://bugs.llvm.org/ and perhaps attach an example of some bitcode that causes the error/assertions?
cheers, sam
On Sat, Apr 13, 2019 at 12:04 AM zhuowei notifications@github.com wrote:
I spent the past week looking into this: it looks like LLVM/WebAssembly's support for custom data sections isn't mature yet, so Swift's runtime type reflection won't work.
I got to the same stage https://github.com/swiftwasm/swift/tree/swiftwasm as @patcheng https://github.com/patcheng's and @MaxDesiatov https://github.com/MaxDesiatov's ports: specifically, I can compile a simple Swift function that doesn't use the stdlib and run it in Firefox.
[image: swiftwasm_first] https://user-images.githubusercontent.com/704768/56073919-515f7f00-5d5f-11e9-9350-daa76c153a4f.png
but I'm stuck here now as well.
I had to comment out a lot of stuff https://github.com/swiftwasm/swift-llvm/tree/stable-swiftwasm in LLVM to get asserts to stop firing when writing the type information sections to the wasm file.This results in invalid wasm files when I compile any Swift file with a class or struct declaration in it... including the stdlib.
While it's possible to disable runtime type information in Swift (with the -disable-reflection-metadata flag), this would break pretty much everything in the stdlib. (I remember from the Swift 2 era that even println("hello world") would crash without runtime type info)
I know @patcheng https://github.com/patcheng sent some patches to LLVM https://reviews.llvm.org/people/revisions/16900/ to try to resolve this, but I'm not sure what they do or whether more changes are needed.
I wonder how other language with runtime types support, such as Go or Rust's Any trait, encode their type information in WebAssembly.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/emscripten-core/emscripten/issues/2427#issuecomment-482783300, or mute the thread https://github.com/notifications/unsubscribe-auth/AAfe5fpT2lriWLn623BjOAw8EOEhmdRsks5vgYF8gaJpZM4CEPAq .
@sbc100 I don't have a Bugzilla account on LLVM yet; I'll apply for one soon.
In the meantime, here are the issues I ran into, with links to patches/workarounds. I'm planning to send some of these upstream when possible
Swift has a bunch of data relocations that are (symbol - constant value); this becomes a MCBinaryExpr, which causes asserts when writing object.
patcheng has a patch to check for these at https://reviews.llvm.org/D42564; I have a terrible kludge for fixing them at https://github.com/swiftwasm/llvm-project/commit/d0a183f5f728d671a453ebba0ecb8a3241566b4f and https://github.com/swiftwasm/llvm-project/commit/3dc1a433498656191d3a2b58c84be91b38ec82cf
This one can be worked around by not passing the equivalent of that flag when compiling Swift
Comdat only works when custom sections aren't specified. Comdat in LLVM's Wasm object writer expects unique section name for each comdat symbol; The default section names are ".rodata.(symbol name)", which does work. If I specify a custom name, it doesn't: https://gist.github.com/zhuowei/597b77698634aac9b302d43b5c464838
Worked around by not using Comdat (since Swift disables comdats on Linux/ELF and macOS anyways)
This isn't an LLVM issue; it's just that Swift writes relative addresses in the metadata table to avoid runtime relocations (ie (target address - current address)); this requires a relocation type referencing two symbols, which doesn't exist in Wasm.
I asked a Swift developer on Twitter and it seems the Windows port has some workarounds for this that I could look into.
Patcheng has a patch for this: https://reviews.llvm.org/D42233 but it predated custom sections support, so it was decided to wait for custom sections to be included and switch the patch to use that.
I've addressed the comment and switched that patch to using custom sections: my version of that patch is https://github.com/swiftwasm/swift-llvm/commit/de2966bb5b0b95639b028c8414b0fa30fc60d1cb and https://github.com/swiftwasm/swift-llvm/commit/502e7536efe71996d920058a3026f304efed4971
ddunbar added Swift calling convention support to llvm, but it isn't turned on yet (maybe because there are no tests?)
I enabled it at https://github.com/swiftwasm/llvm-project/commit/ea5605adc66575b6d88c585febc30121cb098b11, and I'm currently trying to port the ARM32 swift-return.ll/swifterror.ll/swiftself.ll test cases to WebAssembly so I can submit that patch.
Thank you so much for your work on the WebAssembly backend!
@MaxDesiatov A quick update on my progress:
The Stdlib links. ...For certain values of "links": I basically completely broke the metadata support to get it to link. Attempting to run the resulting WebAssembly file gives me:
zhuowei@zhuowei-laptop:~/Documents/wasmtime$ target/release/wasmtime ~/swift-source/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/hello.wasm
failed to demangle superclass of _ContiguousArrayStorage from mangled name 's28__ContiguousArrayStorageBaseC'
error while processing main module /home/zhuowei/swift-source/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/hello.wasm: Instantiation error: Trap occurred while invoking start function: wasm trap at 0x7fbe9489677d
Yeah, this needs a lot more work before swift_getTypeByMangledName would run.
All my changes are pushed to the SwiftWasm repo, and I'll setup CI and document how to build it soon.
@zhuowei - I'm curious to hear more about your goals for the port. You're using wasi, so is this mostly for non-browser use cases? Is running on the web not a goal? (And, is using an API like OpenGL not a goal?)
@kripken Thank you so much for your comments!
I'm looking to target the browser, with serverside being a nice to have.
I mostly used WASI because it's new and defaults to using LLVM's wasi-ld for linking, instead of Emscripten's bitcode-based emcc linker. (I know Emscripten can use wasm-ld as well, but I think? the standard libraries are all compiled for emcc?) I actually started with a Emscripten sysroot, and it didn't take too many changes to switch to WASI. Thus, I'm sure most of the work done for this port would also be applicable to an Emscripten-based port.
I was planning to modify WASI's browser polyfill for better interop with web APIs and JavaScript (since Emscripten's EM_ASM macros probably won't work in Swift anyways). I understand that Emscripten's the more mature choice right now for interoperating with the Web. I would really appreciate your expertise in what approach I should take to turn Swift into a language for the web.
Hi team, playing around with Emscripten in earnest for the first time, not my usual area of work so apologies if this is clearly never going to work. What I'd like to do is compile a simple Swift program to JS:
So far what I've tried is:
but that fails because it can't find certain symbols (looks like the standard library isn't present):
Drilling a bit further, and looking at how
xcrun swift -v hello_world.swift
actually works, I'm able to use this linking command to compile the bitcode to an executable using normalld
.So, it seems like
libswift_stdlib_core.dylib
is the only dependency forhello_world.bc
to be properly linked to an executable.But I'm stuck - is there some equivalent between the
-L
and-rpath
flags onld
that I should be passing toemcc
? Or is a dylib like that not possible to be used in emscripten? The final command I tried was:Hopefully something's possible from here - Swift is a neat high-level language with a nice type system and no garbage collector, which makes me think it's a good fit for compiling to JS.