Cant compile to WASM

[Overtorment]

Ubuntu 18, clang 10 CC=clang-10 wasm-pack build --dev -- -v fail with

cargo:warning=In file included from bitcoin/src/utilstrencodings.cpp:6:
cargo:warning=bitcoin/src/utilstrencodings.h:13:10: fatal error: 'string' file not found
cargo:warning=#include <string>
cargo:warning=         ^~~~~~~~

similar on latest macos

cargo:warning=depend/secp256k1/contrib/lax_der_parsing.c:7:10: fatal error: 'string.h' file not found
cargo:warning=#include <string.h>
cargo:warning=         ^~~~~~~~~~
cargo:warning=1 error generated.

Not sure whose fault is that, maybe this should be reported to wasm-pack or clang is not configured right..?

[TheBlueMatt]

The second looks like an issue with rust-secp (is it https://github.com/rust-bitcoin/rust-secp256k1/issues/134 ?), the first looks like its puling in rust-bitcoinconsensus when building for WASM, which is only for tests, so you could skip building the tests suite at least initially and see if it helps.

[Overtorment]

How do I skip building the tests? I had no idea I was building them

On Mon, 13 Apr 2020 at 17:56, Matt Corallo notifications@github.com wrote:

The second looks like an issue with rust-secp (is it rust-bitcoin/rust-secp256k1#134 https://github.com/rust-bitcoin/rust-secp256k1/issues/134 ?), the first looks like its puling in rust-bitcoinconsensus when building for WASM, which is only for tests, so you could skip building the tests suite at least initially and see if it helps.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rust-bitcoin/rust-lightning/issues/584#issuecomment-612986645, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOTD6L452PQQ6G76PT4DLTRMM72TANCNFSM4MHB2J6Q .

[TheBlueMatt]

Probably --dev? rust-bitcoinconsensus is a dev-dependency.

[TheBlueMatt]

See my comment at https://github.com/rust-bitcoin/rust-secp256k1/issues/134#issuecomment-613122723 . I wasn't able to get wasm-pack to work directly, but cargo build with wasi worked great pretty easily.

[TheBlueMatt]

As for the issue with bitcoinconsensus I guess I was a bit confused as to how dev-dependencies worked. Rust seems to be a bit confused here, but it is really excitedly adding the feature flags listed there even when it doesn't need to. If you build another library which depends on rust-lightning with wasm-pack, however, the issue should go away.

[Overtorment]

okay, I peeked at https://github.com/elichai/rust-lightning/tree/wasm/wasm-test and made it compile to wasm and run it on nodejs. now as fas as I understand there are zero wasm_bindgen in sources so it is not directly consumable by nodejs yet..?

[arik-so]

That's correct. We're more focused on building C-bindings as those can be consumed not only from Node, but from a variety of other languages, too. C-bindings would enable calling RL from Python, Swift, Java, and other environments.

[Overtorment]

can I expect wasm bindings anytime soon? how hard is it to make them? looks like I can make a simple script to parse all sources to prefix all fun and struct with #[wasm_bindgen]

or, if there are C bindings, can they be exposed to javascript somehow?

[arik-so]

A lot of methods can trivially be exposed to WASM the way you describe, but there are types that cannot be, requiring customization. That's why we decided that if customization is required anyway, we should focus on C bindings, which are more versatile. That versatility does include support for Node, too. Those bindings aren't exactly plug and play from the Node side, however (unlike WASM), and require the ffi package, so I'm trying to build some libraries on the host languages' side for interfacing with the auto-generated C bindings. As far as just the C bindings go, it's still at an incipient stage (see here: https://github.com/arik-so/rust-lightning/tree/c_ffi/bindings), but I'm hoping to get something more usable going over the course of the next few weeks.

[Overtorment]

whats the scope to do wasm bindings? in other words, how much work is it? whats the bare minimum to cover with bindings to make at least some PoC with nodejs? like, connect, create and fund the channel, close? how "dumb" is this work? can it be outsourced to someone like me with very limited rust knowledge?

[ariard]

@elichai when you were playing with RL in the browser did you use any any wasm bindings stubs to interact with it :) ?

[Overtorment]

well, @elichai wrote some boilerplate rust code and exported only one function to wasm: https://github.com/elichai/rust-lightning/blob/wasm/wasm-test/src/lib.rs

[elichai]

Yes, So wasm's implementations has sadly changed since then, both in clang and in rust. the problem you're reporting is basically a missing libc for the wasm32-unknown-unknown. target. that libc was removed from clang since clang8. so even my example won't compile today.

I think this can be solved by self compiling libc for that target and passing in a -sysroot flag to CFLAGS, I think the reason clang no longer ships with it is that this target doesn't have any standard for sysroot so shipping with it will guarantee UB across compilation units with different compiler / different versions of the compiler.

So the status of rust+c for wasm is currently sadly pretty bad, emscripten targets were the only really standardized ones but they're currently broken on rust stable.

I haven't really diven deep into the beast of how can you solve this because I don't really have any need for wasm right now, but you can see these issues for more information: https://github.com/rustwasm/team/issues/291 https://github.com/alexcrichton/cc-rs/issues/378 https://bytecodealliance.github.io/wasmtime/wasm-c.html https://github.com/rust-lang/rust/issues/66916 https://lists.llvm.org/pipermail/llvm-dev/2020-February/139276.html

You can also find more information in the discussions in these logs: http://gnusha.org/rust-bitcoin/2020-02-17.log

[TheBlueMatt]

Isn't that the point of wasi? wasi is a libc for wasm, which rustc/cargo seem to support natively.

[elichai]

Isn't that the point of wasi? wasi is a libc for wasm, which rustc/cargo seem to support natively.

I think wasi is for wasm outside of the web? anyway here in the docs of bytecodealliance(they wrote wasm) they're providing pre-built sysroots https://bytecodealliance.github.io/wasmtime/wasm-c.html you can see here: https://github.com/WebAssembly/wasi-sdk/releases wasi-sysroot-10.0.tar.gz

Rust installs a libstd of rust for wasi when you install the target https://forge.rust-lang.org/release/platform-support.html but not a libc

[Overtorment]

this is all very nice, but.

can I expect wasm bindings anytime soon? how hard is it to make them? looks like I can make a simple script to parse all sources to prefix all fun and struct with #[wasm_bindgen]

or, if there are C bindings, can they be exposed to javascript somehow?

[elichai]

I'm not sure what do you mean. This is a library, usually the wasm stuff is done at the "binary" level. Don't forget that there are a lot of flavors to wasm (as you've seen here) so libraries can't go support all of them, they usually support "regular" rust and then you do whatever you need to make it work.

Also it doesn't quite make sense to put #[wasm_bindgen] on top of all the functions, so you should write your own wrappers at the binary level that do as much logic as possible in rust and then export that logic as #[wasm_bindgen].

[Overtorment]

ok, do you think this is a good starting point?

https://github.com/TheBlueMatt/rust-lightning-bitcoinrpc/blob/85a7f6b42ae4494a85138f8d0b2199784fac75c3/src/main.rs#L571

i could refactor those into separate functions and cover them with wasm bindings, then compile it as a lib and see if it works

[TheBlueMatt]

Oof, right, so I hadn't noticed https://github.com/rust-bitcoin/rust-secp256k1/pull/203 which really hurts. Turns out WASM has changed a bunch out from under us wrt C dependencies. Luckily, the changes are all pretty irrelevant for our specific use (libsecp doesn't actually ever call "real" C standard library functions), but we have to do some simple tweaks to make it work again. I got rust-secp working in WASM again this morning (see https://github.com/rust-bitcoin/rust-secp256k1/pull/208), but that may take a week to land and us bump our dependency version. In the mean time, the wasi hack I posted above may work just fine, depending on your use-case, but wasm-pack should start working again after we bump our secp dependency.

As for how to proceed, I pretty much disagree with @elichai here. For proper use of rust-lightning from JavaScript, we really do want to sprinkle #[wasm_bindgen] around liberally so that all of the library is available, maybe with some JavaScript wrappers to make the API feel more "JavaScript-native". I'm not actually sure what the bindgen limitations are (CC @arik-so), so that may not be entirely practical, but a few C-bindings here and there should help address that.

[TheBlueMatt]

Gonna close this. Work is ongoing to export the full library to JavaScript via WASM and is slowly making progress at https://github.com/lightningdevkit/ldk-garbagecollected