search

bitcoindevkit / bdk

A modern, lightweight, descriptor-based wallet library written in Rust!
Other
867 stars 312 forks source link

WebAssembly support or documentation #1671

Open darioAnongba opened 2 weeks ago

darioAnongba commented 2 weeks ago

BDK website states the following:

Cross Platform The core logic is feature rich allowing you to build any wallet from scratch on mobile, desktop and even WebAssembly.

Yet, when trying to use BDK in the browser or get some information about BDK and WebAssembly, I wasn't able to find anything on the documentation and looking into past issues, I was able to understand that some people were successful in compiling bdk to WASM. Yet these issues are years old for most and hard to navigate. I also found the bdk-wasm project from @notmandatory but this project is 2 years old and not up-to-date.

I understand that compiling to WASM is not similar to bindings and is more akin to targeting a specific environment but wouldn't a bdk-wasm repo be interesting as part of the BDK org? Or at least a tutorial section in the documentation?

I'll be happy to contribute to the project and create such repo if you deem it worth it, also writing the documentation for it.

Use case
Being able to use BDK in Node, browser or other JS environments.

Additional context I'm personally trying to use BDK in a browser extension and having trouble obtaining a working package. My knowledge of Rust and WASM is limited and I still have much to learn.

notmandatory commented 2 weeks ago

Hi, thanks for opening an issue for this. I agree we do need a new updated WASM example. Have you taken a look at the https://github.com/MutinyWallet project to see how they used BDK 1.0 with WASM? If you can get a simple example it'd be a good addition to this repo. Or would be great to add WASM tutorial docs to the book of bdk documentation. https://github.com/bitcoindevkit/book-of-bdk

darioAnongba commented 2 weeks ago

Hi @notmandatory, I'll be happy to contribute and write a tutorial on the book-of-bdk.

WASM being a target, it needs to adapt to the env so, as you know better than me, we can only show examples. Nevertheless an "official" example also used to test if the BDK compiles successfully to WASM seems a good idea so a dedicated repo seems a good idea to me.

In the meantime, for those looking to compile to WASM for browser, this is an example of what you need using wasm-pack:

[package]
name = "bdk-wasm"
...

[lib]
crate-type = ["cdylib", "rlib"]

[features]
default = ["console_error_panic_hook"]

[dependencies]
wasm-bindgen = "0.2.95"

# The `console_error_panic_hook` crate provides better debugging of panics by
# logging them with `console.error`. This is great for development, but requires
# all the `std::fmt` and `std::panicking` infrastructure, so isn't great for
# code size when deploying.
console_error_panic_hook = { version = "0.1.7", optional = true }
getrandom = { version = "0.2.15", features = ["js"] }
ring = { version = "0.17.8", features = ["wasm32_unknown_unknown_js"] }
bdk_wallet = { version = "1.0.0-beta.5" }
bdk_esplora = { version = "0.19.0", features = ["async-https"] }

[dev-dependencies]
wasm-bindgen-test = "0.3.45"

[profile.release]
# Tell `rustc` to optimize for small code size.
opt-level = "s"

For MacOS M1/2/3 users, you will need to install llvm and point to it as the default C compiler cannot target WASM and fails for secp256k1:

brew install llvm

and

# In .cargo/config.toml for your project or globally in your `~/.zshrc`
[env]
AR = "/opt/homebrew/opt/llvm/bin/llvm-ar"
CC = "/opt/homebrew/opt/llvm/bin/clang"
darioAnongba commented 2 weeks ago

Hi @notmandatory, it seems the struggle continues.

rusqlite cannot be compiled to wasm_unknown_unknown because it fails onstdio.h:

The following warnings were emitted during compilation:

warning: libsqlite3-sys@0.28.0: sqlite3/sqlite3.c:14605:10: fatal error: 'stdio.h' file not found
warning: libsqlite3-sys@0.28.0:  14605 | #include <stdio.h>
warning: libsqlite3-sys@0.28.0:        |          ^~~~~~~~~
warning: libsqlite3-sys@0.28.0: 1 error generated.

error: failed to run custom build command for `libsqlite3-sys v0.28.0`

Which would make sense since we cannot write files in the browser environment (and even less in a browser extension). Sadly, it seems that this prevents using Connection::open_in_memory() as well even though we are not writing to disk.

So now I'm blocked as I couldn't figure out how to use bdk_wallet without rusqlite. Ideally what I would need is being able to use the memory for state storage and extract it to persist somewhere else.

notmandatory commented 2 weeks ago

thanks @darioAnongba. Did you also see we do have a WASM test in our github CI: https://github.com/bitcoindevkit/bdk/blob/757d83f1076a6267ff5c32430608c8447c348adb/.github/workflows/cont_integration.yml#L85-L112

notmandatory commented 2 weeks ago

I'm focused right now on getting the final 1.0 release out. But after that I'll be happy to review any WASM tutorial docs for the book of bdk.

darioAnongba commented 2 weeks ago

Ok sorry I missed these tests! It reduces the need for a bdk-wasm repo. Still I see you don't try to compile with async-https and rustqlite, which created issues for me.

Any idea for rustqlite? Sadly the WASM rabbit hole is too immense and I can't seem to pass through the errors. Shouldn't the default behaviour of bdk_wallet be an in memory storage ? Am I missing something ?

notmandatory commented 1 week ago

Since WASM doesn't have file system access your best bet is probably to implement your own data store that implements the WalletPersister trait.

riverKanies commented 1 week ago

Hi @darioAnongba 👋

I'm taking a look at this. I pulled your bdk-wasm repo and built this repo to test it out https://github.com/riverKanies/example-bdk-wasm

I have a few questions:

I'm also new to rust and wasm. Would love to get a great bdk-wasm example with solid documentation as I am primarily a web developer. Let's collaborate on this if you're still interested

(btw the sync function isn't fully working for me yet)

darioAnongba commented 1 week ago

Hi @riverKanies and thanks for checking the bdk-wasm repo. The repo is still not working and I am going to continue working on it actively from today. Happy to collaborate on this!

how are you testing it? did you already have a repo that pulls it as a dependency?

I am working on a browser extension and I link it like such for now "bdk-wasm": "file:../../../bdk-wasm/pkg" In terms of pure tests, you have the tests folder of the repo itself. You can run wasm-pack test --headless --firefox/chrome/safari

I was pretty confused about the wasm init function, seems it's common to export an init function in wasm projects but this one doesn't explicitly do so

I think init will be necessary at some point to call set_panic_hook at least. Didn't need it for now.

I found this one from ProtonWallet that seems pretty solid but not up to date with BDK betas (sill uses alphas)

https://github.com/ProtonWallet/andromeda

Some bugs and issues I found already when working with WASM:

I can open separate tickets for these if needed.

riverKanies commented 1 week ago

@darioAnongba great, hopefully we can get a good example working!

I take it your browser extension is not open source? no worries, that's why I made the example-bdk-wasm repo.

Idk how you were able to get wasm running without init, it was necessary for me and I thought it was necessary for all wasm, but maybe browser dependent? what browser are you building the extension for?

comments on the bugs you mentioned:

darioAnongba commented 1 week ago

@riverKanies. Better to keep the discussion here specific to bdk and avoid talking about issues we may have on bdk-wasm (like sync or custom persister).

The browser extension is https://github.com/MetaMask/metamask-extension. Better indeed to try the bdk-wasm on your own example repo.

you're saying esplora is the only supported client?

Sadly yes, Electrum client and RPC clients fail to compile to WASM. Which is important to fix IMO as most users run Electrum servers like electrs.

notmandatory commented 1 week ago

The reason electrum and RPC aren't supported is they use TCP which isn't supported in WASM. The only communications that is supported in WASM is HTTP(s) which is what esplora uses.

darioAnongba commented 4 days ago

Hi @notmandatory, is there a specific reason Electrum uses TCP? wouldn't it be a good idea to support HTTP(s) so that Electrum servers can be called from the browser in the same way as Esplora did?

notmandatory commented 4 days ago

The esplora protocol is already a modernized version of the electrum protocol with essentially the same functionality and it uses HTTP(s).

darioAnongba commented 4 days ago

I understand, I need to open a ticket directly at electrs then. It seems that Esplora on light mode is still around 300GB so people running their own nodes still run electrs.

For example, I'm running my own node using Umbrel and the indexer on the Umbrel Store is electrs. Meaning I cannot connect to my own infra from the browser or browser extensions. Am I missing something?

PS: Ok I deep dived into this matter and opened a discussion here https://github.com/romanz/electrs/issues/820#issuecomment-2482471786 and https://github.com/romanz/electrs/issues/904#issuecomment-2482511989 @cryptoquick was already working on this and it seems he found a solution using mempool API, that is a fork of esplora-electrs as well so works with BDK Esplora client.