Open chadoh opened 1 year ago
Hey @chadoh. thank you for your proposal. This was discussed multiple times before and we don't want to embed the schema into the Wasm for various reasons:
cargo schema
aliasIt would be much better to either let wasmd store the schemas or have them off-chain.
makes the generated Wasm files smaller, rather than larger
It would be interesting to understand why this is the case. If you tested cw-plus then probably because you compared cosmwasm/workspace-optimizer:0.12.8 (Rust 1.63) with cosmwasm/workspace-optimizer:0.12.11 (Rust 1.66).
honestly not sure why there are two scripts in the first place; we can do it all with Rust, get rid of the bash scripts, and make it check if root Cargo.toml defines members or not
Historical reasons. I'd be very happy to explore if this can be consolidated to one builder image.
Thanks for the quick response, @webmaster128! Even on a draft.
And sorry to re-open a discussion that has happened before. I talked to a few people in the Cosmos ecosystem and none of them were aware of previous discussions about this, and they all liked the idea. Is there somewhere I can read the previous discussions?
I actually realized some errors in my original post and have updated it. Re: "makes the generated Wasm files smaller, rather than larger" – that's only somewhat true. While testing locally, I was only running the build_workspace
script, and forgot that it does not run the wasm-opt
step. After running wasm-opt
, I expect that the schema'd Wasm files will be somewhat larger. I can get you good numbers on that tomorrow.
Larger Wasm files increase compile cost and the module loaded into memory when instantiated
Yes, a little. Is it worth it, though?
I'll get you numbers on how much space this adds. With optimized contract sizes on the order of 100-200kB, adding <10kB to a custom section containing the schema doesn't seem bad to me.
And the benefits are huge! Building in the CosmWasm ecosystem will become 10x easier.
5-10% "bloat" for 10x dev experience seems worth it, imo.
I can see the argument for storing them off-chain. How responsive is IPFS these days? That could be a solution. But it will slow down the optimize process, as well as the fetching-of-schemas. Given the small size of the embedded schema, I'm not sure it's worth it.
It's hard to find this data
Not sure what you mean by this.
Not all contracts use the
cargo schema
alias
Yes, and I actually didn't use it here, either. I used cargo run --bin schema
😛
The tongue-out face because I assume not every project implements a schema
binary, either! And yeah, thanks for the reminder, I added a TODO item for this. It definitely needs to deal gracefully with such projects, and just not worry about building or injecting the schema.
The schema format is not necessarily stable
Thanks again for the reminder! I added another TODO item for this. The embedded schema absolutely needs to be versioned, so that the schema format can continue to evolve, independently of the tooling that consumes it.
Here's the current output when building the cw-plus
repo:
Found workspace member entries: ["packages/*", "contracts/*"]
Package directories: ["contracts/cw1-subkeys", "contracts/cw1-whitelist", "contracts/cw20-base", "contracts/cw20-ics20", "contracts/cw3-fixed-multisig", "contracts/cw3-flex-multisig", "contracts/cw4-group", "contracts/cw4-stake", "packages/controllers", "packages/cw1", "packages/cw2", "packages/cw20", "packages/cw3", "packages/cw4"]
Contracts to be built: ["contracts/cw1-subkeys", "contracts/cw1-whitelist", "contracts/cw20-base", "contracts/cw20-ics20", "contracts/cw3-fixed-multisig", "contracts/cw3-flex-multisig", "contracts/cw4-group", "contracts/cw4-stake"]
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
Compiling cw1-subkeys v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys)
Finished release [optimized] target(s) in 9.44s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
Compiling cw1-subkeys v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys)
Finished dev [unoptimized + debuginfo] target(s) in 1.25s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys/schema/cw1-subkeys.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys/schema/cw1-subkeys.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw1_subkeys.wasm
compressed schema size: 3kB
original Wasm size: 331kB
Wasm with injected schema: 314kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
Finished release [optimized] target(s) in 5.67s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
Finished dev [unoptimized + debuginfo] target(s) in 0.66s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist/schema/cw1-whitelist.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist/schema/cw1-whitelist.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw1_whitelist.wasm
compressed schema size: 2kB
original Wasm size: 216kB
Wasm with injected schema: 204kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.05s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base/schema/cw20-base.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base/schema/cw20-base.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw20_base.wasm
compressed schema size: 3kB
original Wasm size: 367kB
Wasm with injected schema: 349kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.05s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20/schema/cw20-ics20.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20/schema/cw20-ics20.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw20_ics20.wasm
compressed schema size: 2kB
original Wasm size: 390kB
Wasm with injected schema: 367kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.05s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
Finished dev [unoptimized + debuginfo] target(s) in 0.83s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig/schema/cw3-fixed-multisig.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig/schema/cw3-fixed-multisig.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw3_fixed_multisig.wasm
compressed schema size: 6kB
original Wasm size: 343kB
Wasm with injected schema: 327kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
Compiling cw3-flex-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig)
Finished release [optimized] target(s) in 16.43s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
Compiling cw3-flex-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig)
Finished dev [unoptimized + debuginfo] target(s) in 1.36s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig/schema/cw3-flex-multisig.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig/schema/cw3-flex-multisig.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw3_flex_multisig.wasm
compressed schema size: 6kB
original Wasm size: 438kB
Wasm with injected schema: 417kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.06s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group/schema/cw4-group.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group/schema/cw4-group.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw4_group.wasm
compressed schema size: 0kB
original Wasm size: 242kB
Wasm with injected schema: 226kB
Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake"...
1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished release [optimized] target(s) in 0.05s
2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake/schema/cw4-stake.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake/schema/cw4-stake.json
3. inject compressed JSON into Wasm
updating in place: target/wasm32-unknown-unknown/release/cw4_stake.wasm
compressed schema size: 2kB
original Wasm size: 295kB
Wasm with injected schema: 278kB
Another complication with using something like IPFS or other off-chain storage is that uploading contracts requires "logging in" / adding keys to two different systems: the target blockchain and the schema-storage location.
Hey @chadoh. thank you for your proposal. This was discussed multiple times before and we don't want to embed the schema into the Wasm for various reasons:
- Larger Wasm files increase compile cost and the module loaded into memory when instantiated
is this already compressed with the best tool for compression? We were wondering if we can use gzip
or something to reduce it further?
updating in place: target/wasm32-unknown-unknown/release/cw1_subkeys.wasm
compressed schema size: 3kB
original Wasm size: 331kB
Wasm with injected schema: 314kB
- It's hard to find this data
would there be an way to access this and/or make an API endpoint to get the schema so it's not hard to find?
- Not all contracts use the
cargo schema
alias
maybe we can make it a part of the build so it's always run and developers don't have to manually create schemas. Seems that abstracting this step from rust devs could be a good thing.
- The schema format is not necessarily stable
Since we now have IDL versions, we can always detect the schema version. If it changes, the tooling can know how to parse the schema based on the IDL which is in the schemas now.
cc @ethanfrey this is the PR I was mentioning
honestly not sure why there are two scripts in the first place; we can do it all with Rust, get rid of the bash scripts, and make it check if root
Cargo.toml
definesmembers
or not
This was now changed and will be shipped with the next release 0.15.0. Then we only have one image for x86_64 and one for ARM64.
Wouldn't it be great to always have schemas available right on-chain?
Then tooling can plug right in. Tools like ts-codegen can build TS wrappers for on-chain contracts. Frontends like RAEN Admin can generate interactive docs. You can build interactive CLIs for specific deployed contracts. Imagine!
Perhaps this hasn't been done out of fear of bloating the size of deployed Wasm files. But if we compress the JSON with brotli, it ends up adding just a tiny number of kB for most contracts. The largest compressed schemas in the cw-plus repo weigh in at 6kB, for the multisig contracts. That's 6kB on (uncompressed) contracts weighing in at 343kB (fixed multisig) and 438kB (flex multisig). Not bad!
Since the point here is to only keep around schema'd Wasms, I just decided to update in-place. We could keep the old un-schema'd files around, though, if y'all prefer.
To do
Cargo.toml
definesmembers
or not).schema
binary / alias. In these cases, just don't add the schema.