Substreams Sink to KV (gRPC/Connect-Web API)

[maoueh]

Now that we have a Substreams sink to KV (https://github.com/streamingfast/substreams/issues/93), we will know try to have something easier for the developers to consume the KV store.

We are going to slap a gRPC + Connect Web.on top of the substream-sink-kv. At the infrastructure level, we expect a single binary to run that will do 2 things:

• Continuously sink the KV with on the storage engine
• Serve through a Connect-Web (HTTP) interface proxy to a gRPC server serving a low-level API on top of the sinked key/values.

rpc {
  // Proper documentation for all request, also describes what happen on error cases (for example, on XXX it returns `InvalidArgument` gRPC code!
  Get(GetRequest) GetResponse
  GetMany(GetManyRequest) GetManyResponse
  Scan(ScanRequest)…
  Prefix(PrefixRequest)…
}

message GetResponse {
  any.Any value = 1;
}

message ScanRequest{
  uint64 limit;
  ...

And the payload would simply be an any.Any, with the value of the key, and the client would do the decoding, because he knows what he’s querying.

While doing this don't forget:

• [x] Create a small example pure HTML/JavaScript that consumes the Connect-Web interface
• [x] Document the gRPC interface properly with details about what it does but especially the error paths.
• [x] Write down important running information about the project (you need to run this, and this, and that, blah blah)

Design should allow these examples substreams to be built

• https://github.com/streamingfast/substreams/issues/136 TokenBalance
• https://github.com/streamingfast/substreams/issues/137 NFT history

[sduchesneau]

What do we do with maximum results and pagination for requests such as "Scan" and "Prefix" ? Without "time-traversal" properties on the kv store, we will end up with the same design problem as eth rpc calls...

[sduchesneau]

Small analysis of concurrency problems:

1. on Badger (local DB only), we can implement WriteLock around the operations of a block
2. on TiKV, there are ways of wrapping our changes in a transaction (then make it "one block == one transaction"
3. on BigTable, there are no transactions or decent locking possibility, but we could use the timestamps of cells to filter by block number: https://cloud.google.com/bigtable/docs/using-filters#timestamp-range

on our own 'netkv', I don't see a good solution..

[maoueh]

Reopening because the task will be closed when instructions will have been sent to Pinax and our Discord

[maoueh]

Also, here a few things that are missing:

• [x] Proper release to GitHub version 0.1.0 of the binary + README update for instructions (fix indentation of first instruction also)

• [x] Some minor update to README.md:

  • Note or Warning should be formatted with new GitHub Note/Warning syntax (see raw source of this comment for raw markup):

    Note This is an indented note

    Needs a paragraph in between

    Warning And a warning

  • Run command authentication note should also show error message that is displayed when not authenticated, the note in itself could be clearer.
  • (these commands have been tested while running the 'devel/local/start.sh' in another terminal) should be in a note after the commands. Same thing for browser note (this has been tested while running the 'devel/local/start.sh' in a terminal)
  • Cargo crate heading is wrong, it's bigger than others, also, the section should be improve to better explain how to add it and use it, something quick
• [x] Tutorial in the documentation about all this, substreams-sink-files tutorial currently in PR can be used
• [x] Instructions is sent to Pinax about the tool ready to be tested
• [x] Discord quick announcement also if devs want to test it, it should be quick and point to tutorial in documentation and README notes.

[sduchesneau]

@maoueh please review https://substreams.streamingfast.io/developers-guide/substreams-sinks/substreams-sink-kv#consume-the-key-value-data-from-a-web-page-using-connect-web

then we can close this issue :D

[maoueh]

All good, just some alignment of code blocks with bullet list, closing anyway so we move on.