feat: adding `llvm` and `clang` to base-optimizer for supporting contracts with FFI

[SebastianElvis]

Problem

We have been working on a Wasm smart contract that might use rust-bitcoin as a dependency. The secp256k1 functionalities in rust-bitcoin use FFI to some C++ code, thus require llvm-ar and clang binaries for compiling the Wasm smart contract as discussed here. The final rust-optimizer image omits these binaries, thus cannot compile such Wasm smart contracts with FFI.

This PR

This PR proposes a fix to this issue. It adds clang and llvm to the base-optimizer target, such that rust-optimizer and workspace-optimizer will inherit them and can compile Wasm smart contracts with FFI.

[webmaster128]

Which secp256k1 functions do you need? Can't you use CosmWasm crypto APIs for things like secp256k1 signature verification?

[SebastianElvis]

Which secp256k1 functions do you need? Can't you use CosmWasm crypto APIs for things like secp256k1 signature verification?

Thanks for your quick response.

So we are building an on-chain BTC light client in Wasm smart contracts. The functionalities we expect to implement do not involve secp256k1. The issue is that when importing rust-bitcoin (since we want to reuse some logic there), the compiler will compile rust-bitcoin and its dependencies, including rust-secp256k1. The rust-secp256k1 uses FFI to the C++ implementation of secp256k1, leading to the LLVM issue.

[webmaster128]

I see 🤔 In general we advise against using dependencies that are not implemented in pure Rust. I'm not sure if it is desired to increase the toolchain shipped here. Is it even possible to link Rust and C/C++ code together for a Wasm build?

Did you try a local Docker build with this change? Does it work for your project?

[SebastianElvis]

I see 🤔 In general we advise against using dependencies that are not implemented in pure Rust. I'm not sure if it is desired to increase the toolchain shipped here.

I totally agree with your concern. We did some research about this and it seems that Wasm provides little support for FFI. I'm also not entirely sure whether using FFI in Wasm smart contracts is a good idea or not so submitted this PR to get more feedback.

Is it even possible to link Rust and C/C++ code together for a Wasm build?

Did you try a local Docker build with this change? Does it work for your project?

Yep we did some experiments about this, and surprisingly it works. Specifically, we injected some code that uses rust-bitcoin to sign and verify a message to a Wasm smart contract, and executed it in Wasm VM. The code compiles and the verification/assertions had passed. The project is still WIP so unfortunately we cannot share it publicly at the moment.

I guess the reason is that rust-secp256k1 has put considerable effort on supporting Wasm targets, evidenced by the bindings and CI tests for Wasm.

[maurolacy]

So we are building an on-chain BTC light client in Wasm smart contracts. The functionalities we expect to implement do not involve secp256k1. The issue is that when importing rust-bitcoin (since we want to reuse some logic there), the compiler will compile rust-bitcoin and its dependencies, including rust-secp256k1. The rust-secp256k1 uses FFI to the C++ implementation of secp256k1, leading to the LLVM issue.

Interesting.

I don't think you'll be able to call those FFI functions from the smart contract context. So, perhaps adding support for them is just opening a can of worms.

Wouldn't be possible to fork rust-bitcoin, and create a 'lite' version of it with just the functionality you need? Perhaps moving that upstream, depending on some feature flags ('rust-runtime' / 'lite-runtime') or so?

[webmaster128]

The other solution I could think of is force-replacing the rust-secp256k1 dependency at project level with a dummy package if it is not used anyways.

[SebastianElvis]

Thanks @webmaster128 and @maurolacy for the helpful feedbacks. So it looks like supporting FFI in Wasm smart contracts may not be a good idea. Closing this PR then.

I don't think you'll be able to call those FFI functions from the smart contract context. So, perhaps adding support for them is just opening a can of worms.

Yep agree that supporting FFI in Wasm smart contracts may not be a good idea. Nevertheless, we actually run rust-secp256k1 code successfully in Wasm VM. Will do more tests.

Wouldn't be possible to fork rust-bitcoin, and create a 'lite' version of it with just the functionality you need? Perhaps moving that upstream, depending on some feature flags ('rust-runtime' / 'lite-runtime') or so?

The other solution I could think of is force-replacing the rust-secp256k1 dependency at project level with a dummy package.

Indeed, this is an alternative approach we are considering. The drawback is that we will need to maintain a fork of rust-bitcoin, so it will come with a bit more overhead. Will investigate more about the overhead.

[webmaster128]

Keep us posted on your findings and progress, maybe in a ticket. If there are compelling reasons to do this change, I would not block it.

[maurolacy]

Yep agree that supporting FFI in Wasm smart contracts may not be a good idea. Nevertheless, we actually run rust-secp256k1 code successfully in Wasm VM.

I wonder how that works. Is the wasm code calling native code / libs? How are those libs going to be packaged / distributed to blockchain nodes?

What about the security implications of that? If I understand correctly, you are basically breaking out of the VM context.

Will do more tests.

Keep us posted on your findings.

[SebastianElvis]

Hi there, I have done some more experiments on the Wasm smart contracts with FFI calls. The result is pretty interesting. If I compile and optimise this contract at my laptop, the contract will fail the check of cosmwasm-check with Unknown opcode "192". But, if I comile and optimise this contract by using the rust-optimizer Docker container with llvm/clang, the contract will pass the check of cosmwasm-check.

So we developed a vanilla Wasm smart contract, added a dependency to rust-bitcoin which has FFI calls to C++ libsecp256k1, and injected the following code to pub fn instantiate (the function invoked when instantiating a smart contract on Cosmos zone):

    let secp = bitcoin::secp256k1::Secp256k1::new();
    let message = b"rust-bitcoin MessageSignature test".to_vec();
    let msg =
        bitcoin::secp256k1::Message::from_hashed_data::<bitcoin::hashes::sha256::Hash>(&message);
    let privkey = bitcoin::secp256k1::SecretKey::from_slice(&[0xcd; 32]).unwrap();
    let pubkey = privkey.public_key(&secp);
    let secp_sig = secp.sign_ecdsa(&msg, &privkey); // this involves FFI call
    let verifiation_result = secp.verify_ecdsa(&msg, &secp_sig, &pubkey); // this involves FFI call
    assert!(verifiation_result.is_ok());

Then, I compiled and optimised the code by using two approaches. The first approach is by executing the following Makefile entry on my Macbook

build-optimized-local-mac:
    @RUSTFLAGS='-C link-arg=-s' AR=/usr/local/opt/llvm/bin/llvm-ar CC=/usr/local/opt/llvm/bin/clang cargo wasm
    @wasm-opt -Os ./target/wasm32-unknown-unknown/release/babylon_contract.wasm -o ./artifacts/babylon_contract.wasm

The compiled contract size is 1.3M. The cosmwasm-check gives me the following result:

➜  babylon-contract git:(btc-utils-fork) ✗ make build-optimized-local-mac && ls -lh ./artifacts && cosmwasm-check ./artifacts/babylon_contract.wasm
   Compiling babylon-contract v0.0.0 (/Users/runchao/Projects/Babylon/babylon-contract)
    Finished release [optimized] target(s) in 13.51s
total 528
-rw-r--r--  1 runchao  staff   198K Mar 24 09:54 babylon_contract.wasm
-rw-r--r--  1 runchao  staff    88B Mar 24 09:43 checksums.txt
-rw-r--r--  1 runchao  staff   128B Mar 24 09:43 checksums_intermediate.txt
Available capabilities: {"cosmwasm_1_2", "stargate", "staking", "iterator", "cosmwasm_1_1"}

./artifacts/babylon_contract.wasm: failure
Error during static Wasm validation: Wasm bytecode could not be deserialized. Deserialization error: "Unknown opcode 192"

Also, even if I removed the rust-bitcoin dependency, the error persists. So this error seems not relevant to the FFI stuff.

The second approach is by executing the following Makefile entry on my Macbook

build-optimized:
    if [ -z $$(docker images -q $(OPTIMIZER_IMAGE_NAME)) ]; then \
        make rust-optimizer-image; \
    fi
    $(DOCKER) run --rm -v "$(CUR_DIR)":/code \
        --mount type=volume,source="$(CUR_BASENAME)_cache",target=/code/target \
        --mount type=volume,source=registry_cache,target=/usr/local/cargo/registry \
        $(OPTIMIZER_IMAGE_NAME):$(OPTIMIZER_IMAGE_TAG)

where the Dockerfile is simply a wrapper of rust-optimizer with llvm/clang added

FROM cosmwasm/rust-optimizer:0.12.12 as rust-optimizer

# clang and llvm are required for compiling rust-secp256k1 in rust-bitcoin
RUN apk update && \
    apk add --no-cache clang llvm

WORKDIR /code

ENTRYPOINT ["optimize.sh"]
# Default argument when none is provided
CMD ["."]

The cosmwasm-check gives me the following result:

➜  babylon-contract git:(btc-utils-fork-test-secp256k1) cosmwasm-check ./artifacts/babylon_contract.wasm
Available capabilities: {"iterator", "stargate", "cosmwasm_1_2", "cosmwasm_1_1", "staking"}

./artifacts/babylon_contract.wasm: pass

All contracts (1) passed checks!

It's also worth mentioning that from my side, if a contract passes cosmwasm-check, then I can store and instantiate the contract on a wasmd Cosmos zone locally, and vice versa.

This was pretty strange since to my understand rust-optimizer basically does the same thing as wasm-opt -Os. The only difference between these two approaches of compilation is the environment (MacOS and rust-alpine). Not fully sure why this affects the cosmwasm-check. Any thoughts or feedbacks are welcome :)

I wonder how that works. Is the wasm code calling native code / libs? How are those libs going to be packaged / distributed to blockchain nodes?

My suspect is that the Rust compiler just packages everything altogether, including the compiled libsecp256k1 written in C++. As long as the code does not involve FFI calls to OS functionalities, the FFI calls will happen inside the Wasm code and do not go beyond the Wasm VM. This is evidenced in the contract size (1.3 M), which is likely to be a consequence of including the entire libsecp256k1 code in the Wasm file.

[maurolacy]

Thanks for the detailed report!

Regarding

This was pretty strange since to my understand rust-optimizer basically does the same thing as wasm-opt -Os. The only difference between these two approaches of compilation is the environment (MacOS and rust-alpine). Not fully sure why this affects the cosmwasm-check. Any thoughts or feedbacks are welcome :)

the answer probably is that the environment (MacOs vs. Alpine Linux) is affecting the end result, yes. An interesting test would be, to try and optimise the contract with the (modified) rust-optimizer-arm64 docker image.

My suspect is that the Rust compiler just packages everything altogether, including the compiled libsecp256k1 written in C++. As long as the code does not involve FFI calls to OS functionalities, the FFI calls will happen inside the Wasm code and do not go beyond the Wasm VM. This is evidenced in the contract size (1.3 M), which is likely to be a consequence of including the entire libsecp256k1 code in the Wasm file.

Interesting. Next question then: What would happen if the FFI call does involve calls to OS functionalities? A simple example / test would be: wrap a call to the system's clock in a native library, and call it through FFI from a wasm smart contract. Are you able to obtain the system's wall time that way? Does this fail at runtime? Does it fail at compile / optimisation time?

[maurolacy]

I wonder how that works. Is the wasm code calling native code / libs? How are those libs going to be packaged / distributed to blockchain nodes?

My suspect is that the Rust compiler just packages everything altogether, including the compiled libsecp256k1 written in C++. As long as the code does not involve FFI calls to OS functionalities, the FFI calls will happen inside the Wasm code and do not go beyond the Wasm VM. This is evidenced in the contract size (1.3 M), which is likely to be a consequence of including the entire libsecp256k1 code in the Wasm file.

I think that's correct. If the C / C++ module / dependency of the Rust crate is prepared to compile for the wasm32 target, the compilation and linking process resolves all the links to ffi code as jumps to wasm code, not native code. Which makes sense, as the compilation target indicates.

I guess calls to OS functionalities from foreign code will be just forbidden by the WebAssembly sandbox implementation / concept. The same as in the Rust case.

Confirming this, either by citing sources or doing experiments will be nice though. More about this later I guess.