[x] Create substreams-sink-kv inject <dsn> <substreams-endpoint> <spkg> --listen-addr 0.0.0.0:8080, with default listen-addr to empty string, in which case the program doesn't setup the server. If --listen-addr is != "", then also start the server.
[x] Create substreams-sink-kv serve --listen-addr=0.0.0.0:8080 <dsn> <spkg> with default listen-addr=127.0.0.1:8080. Don't run injector in this case.
[x] Implement the other KV access functions exposed to the WasmEdge VM (kv_get_key is there, we'll add the many variant, and scan and prefix ones).
[x] Where we decode the spkg, also decode the sink_config and make sure it matches the sf.substreams.sink.kv.v1.WASMQueryService OR sf.substreams.sink.kv.v1.GenericService (see below)
[x] If it's the WASMQueryService, then use the experiment code.
[x] Use the grpcService to route inside the WASM machine, not the hard-coded value we have right now.
[ ] We need to validate that a substreams transformation module with target wasm32-unknown-unknown will be compiled WITHOUT the wasi extensions, even if our crate contains OPTIONAL (in sub-packages like server) code that would compile those wasi_snapshot-preview.fd_write methods). The goal is to have a SINGLE crate called substreams-sink-kv, but that we could use from within substreams modules (when doing transforms), AS WELL as in the KV Sink's inject/serve modes,, which is a totally different Wasm environment. Substreams compilation needs to ignore that server package.
[ ] Document a Quickstart, on how to build your first sf_mycustomer_v1_eth_transfers mapper.
[ ] We need a pack cobra command, that will take that sink-kv and spit out a new spkg, based on the package.
[ ] Use the same conventions as substreams pack to build the output filename, based on the package name and version. Start by copying from substreams, and we'll refactor if we start seeing too many copies of that.
[ ] This pack command needs to pick up protobuf references, and add them to the output spkg file, after importing
[ ] The WasmEdge runtime needs to implement the Connect Web layer: picking up the grpcService from the WASMQueryService definition in the package, and doing the Connect Web dance over the generic gRPC service we already have.
Third stage:
[ ] Build a macro to simplify the Rust code, so you don't need to unpack proto and pack proto on your own.
Protobuf definition for sf/substreams/sink/kv/v1/services.proto:
// This defines a KV Sink to be queried with a generic key access interface (Get, GetMany, Scan, Prefix calls).
message sf.substreams.sink.kv.v1.GenericService {
}
// This defines configuration to run a WASM query service on top of the KV store being sync'd.
message sf.susbtreams.sink.kv.v1.WASMQueryService {
int64 start_block = 1;
// wasm exports: "kv_get_batch", "kv_get", "kv_scan", "kv_prefix"
bytes wasm_query_module = 5;
// Fully qualified Protobuf Service definition name
string grpc_service = 2; // sf.mycustom.v1.MyService
}
This is the design for a WASM query layer
It starts from the expriment branch
feature/dynamic-grpc-test-dont-mergefromsubstreams. Here: https://github.com/streamingfast/substreams/compare/develop...feature/dynamic-grpc-test-dont-mergewasmedgefrom https://wasmedge.org/book/en/quick_start/install.html to README.mdmake build, checking whether ~/.wasmedge/... exists.rustup target add wasm32-wasiat https://wasmedge.org/book/en/write_wasm/rust.htmlsubstreams-sink-kv inject <dsn> <substreams-endpoint> <spkg> --listen-addr 0.0.0.0:8080, with defaultlisten-addrto empty string, in which case the program doesn't setup the server. If --listen-addr is != "", then also start the server.substreams-sink-kv serve --listen-addr=0.0.0.0:8080 <dsn> <spkg>with defaultlisten-addr=127.0.0.1:8080. Don't run injector in this case.kv_get_keyis there, we'll add the many variant, and scan and prefix ones).spkg, also decode thesink_configand make sure it matches thesf.substreams.sink.kv.v1.WASMQueryServiceORsf.substreams.sink.kv.v1.GenericService(see below)grpcServiceto route inside the WASM machine, not the hard-coded value we have right now.substreams transformation modulewith targetwasm32-unknown-unknownwill be compiled WITHOUT thewasiextensions, even if our crate contains OPTIONAL (in sub-packages likeserver) code that would compile thosewasi_snapshot-preview.fd_writemethods). The goal is to have a SINGLE crate calledsubstreams-sink-kv, but that we could use from within substreams modules (when doing transforms), AS WELL as in the KV Sink'sinject/servemodes,, which is a totally different Wasm environment. Substreams compilation needs to ignore thatserverpackage.sf_mycustomer_v1_eth_transfersmapper.sink-kv.yamlmanifest reader, that packs it into the proper protobuf message (either GenericService or WASMQueryService). Sample manifest here: https://github.com/streamingfast/substreams/issues/177#issuecomment-1440958604import,package,moduleSecond stage:
packcobra command, that will take thatsink-kvand spit out a newspkg, based on thepackage.substreams packto build the output filename, based on the package name and version. Start by copying fromsubstreams, and we'll refactor if we start seeing too many copies of that.protobufreferences, and add them to the output spkg file, after importinggrpcServicefrom the WASMQueryService definition in the package, and doing the Connect Web dance over the generic gRPC service we already have.Third stage:
Protobuf definition for
sf/substreams/sink/kv/v1/services.proto: