Upstream featured WASM optimization

[CyberHoward]

The need for featured contract compilation has been thoroughly discussed in issue #80. This PR upstreams the ability to optimize smart-contracts with different features, without hurting reproducibility.

Example

As an example let's say you have some functionality that should only be enabled on some specific chain. Currently it's extremely hard to do this without creating different contracts.

Our solution uses cargo's support for custom metadata fields to allow you to specify features that you want your contract to be built with.

I.e. if I want different logic when deploying to the cw-sdk then I add a feature in the contract's Cargo.toml like so:

[features]
cw-sdk = []

To trigger the optimizer into building the contract with this feature add the following to the same Cargo.toml:

[package.metadata.optimizer]
builds = [
  { name = "cw-sdk", features = [
    "cosmwasm_1_3",
  ] }
]

After running the optimizer image as usual you will be left with:

• my_contract.wasm
• my_contract-cw-sdk.wasm

When using the ARM image:

• my_contract-aarch64.wasm
• my_contract-cw-sdk-aarch64.wasm

Testing

This feature was tested by adding featured-flagged include_bytes!(...) statements in the contracts which results in a difference in .wasm artifacts.

Credit to adair for doing most of the implementation!

[CyberHoward]

I updated the syntax to the following;

[package.metadata.optimizer]
builds = [
  { name = "juno", features = [
    "cosmwasm_1_3",
    "tokenfactory",
  ] },
  { name = "osmosis", features = [
    "cosmwasm_1_3",
    "tokenfactory",
  ] },
  { name = "stargaze", features = [
    "cosmwasm_1_2",
    "governance_gated",
    "tokenfactory",
  ] },
]

I also updated the logic to only compile if it encounters a feature combination that it has not compiled yet. If a feature combination has already been compiled then it will copy and rename the already built WASM to fit the requested name. I.e. in the example above the WASM will be built for juno but the WASM for osmosis will be a copy of the previously built juno artifact.

Any feedback before I start cleaning up and unify the single-contract and workspace optimizers? @webmaster128

[CyberHoward]

I've unified the single-contract and workspace build processes into the cw-build directory.

I think there's something to be said for unifying the rust-optimizer and workspace-optimizer images by simply checking if the Cargo.toml in the provided directory is a workspace or not.

[apollo-sturdy]

Any update on getting this merged?

[webmaster128]

I would like to break this down in smaller steps across multiple PRs, but the overall direction is great. E.g.

• Rust upgrade should always be separate (done on main)
• Let's use a Rust-based build script to avoid having to maintain similar logic in both Rust and Shell
• The build modes proposed here

[webmaster128]

I think there's something to be said for unifying the rust-optimizer and workspace-optimizer images by simply checking if the Cargo.toml in the provided directory is a workspace or not.

Let's work in that direction, but keep it a separate step. This came up before and is getting more and more obvious to do.

[webmaster128]

If a feature combination has already been compiled then it will copy and rename the already built WASM to fit the requested name. I.e. in the example above the WASM will be built for juno but the WASM for osmosis will be a copy of the previously built juno artifact.

This sounds like an unnecessary amount of complexity for this tool. One the one hand you can just do

  { name = "juno_osmosis", features = [
    "cosmwasm_1_3",
    "tokenfactory",
  ] },

in your metadata. On the other hand re-compilation should be cheap with sccache in place. Are you sure we really need to have this feature?

[webmaster128]

Thinking about this more, I really want to avoid the copy-by-features logic. Think of the following extension of that concept: Right now, we apply --signext-lowering to wasm-opt to support chains running CosmWasm < 1.3. One meta field could be signext-lowering (bool, defaults to true) in order to use the new Wasm instructions on newer CosmWasm version. Then you'd use that build config to impact the wasm-opt call. If Juno did the upgrade and Osmosis did not, you end up with the same cargo features but different builds still.

[CyberHoward]

Thinking about this more, I really want to avoid the copy-by-features logic. Think of the following extension of that concept: Right now, we apply --signext-lowering to wasm-opt to support chains running CosmWasm < 1.3. One meta field could be signext-lowering (bool, defaults to true) in order to use the new Wasm instructions on newer CosmWasm version. Then you'd use that build config to impact the wasm-opt call. If Juno did the upgrade and Osmosis did not, you end up with the same cargo features but different builds still.

We could easily expand the copy logic to only copy if the build setting are the same (instead of just the feature set).

[webmaster128]

Yeah, I guess so. But I'm not convinced we need or want to maintain an additional caching layer here.

[webmaster128]

Thanks again for all the great work in here. Let's get those things in one-by-one. In order to be able to break down this work and get something relatively small started, I created #134. I basically takes a lot of the work of the Rust-based build system script from here and implements it. Some key differences:

• Builder project is now called bob_the_builder and the binary is called bob
• The --workspace argument is not needed as bob detects if the folder it is running in is a workspace or not

It does not yet merge the two builder images but provides a framework in which new features can be implemented in Rust.

[webmaster128]

Hey @CyberHoward, following up on my latest comment: On main you find a version in which only one Docker image exists for both single contract and workspace builds. Both cases use a single optimize.sh which delegates the build to bob.

So it would be a great time to pick up this work again and implement the build matrix solution we discussed in the Rust project bob_the_builder.

[apollo-sturdy]

@CyberHoward @webmaster128 Do either of you have time to work on this? We're still using abstractmoney/rust-optimizer in several of our repos and would like to use the official image for better reproducibility.

[webmaster128]

Massive amounts of restructuring work have been done in a series of PRs that were merged to main and are shipped. Right now the build system bob_the_builder is only in Rust and there is only one image for single contracts and workspaces (cosmwasm/rust-optimizer and cosmwasm/workspace-optimizer are now aliases for cosmwasm/optimizer). With that we have a great foundation to do the actual implementation.

From here it would be great to implement the feature as discussed in here. Personally I don't have time to work on it for the initial development but happy to review and maintain it. If there is more preparational work required, please split into minimal independent PRs.

[CyberHoward]

@CyberHoward @webmaster128 Do either of you have time to work on this? We're still using abstractmoney/rust-optimizer in several of our repos and would like to use the official image for better reproducibility.

Yeah I can make some time for it! Will create a fresh PR soon.

[apollo-sturdy]

@CyberHoward @webmaster128 Do either of you have time to work on this? We're still using abstractmoney/rust-optimizer in several of our repos and would like to use the official image for better reproducibility.

Yeah I can make some time for it! Will create a fresh PR soon.

Yay! You da man! <3

[CyberHoward]

Closing in favor of #148