Register table and index schema with schema manager

[aaronc]

Motivation

The merkle store of chains is exposed directly to clients for querying and ideally they can use it for generating merkle proofs. This is only useful if clients actually understand the format of the merkle store. With the ORM package, we have a formal way for organizing the merkle store. With a proper "schema manager" that tracks or even better assigns table and index prefixes to table and index types, a store introspection API could be exposed to clients.

Definition of Done

The "schema manager" should:

• expose an API to clients that describes all key formats and the table/index they correspond to in protobuf format
• automatically assign table prefixes deterministically - that is all validators of the same state machine should end up with the same prefixes

---

Thoughts @alpe?

[alpe]

I do understand the problem but I would need some more information to come up with some concrete ideas. How would the ideal workflow be for the client? Let's use as example: get a list of proposals?

1. I assume they would query the schema manager for the "group" package or is it an application wide one?
2. To access an index or table for example there would be a human readable name and a prefix mapping somewhere.
3. Then the client would use the proper prefix and do a raw query, correct?

I am a bit biased by what we had build for weave. There you would get a list of proposals by querying "/proposals" which is the bucket name. A query to /proposals/author with an address would be to an index. There is no endpoint exposing the query path. They were shared with the clients by application documentation.

[aaronc]

So I think it would help for me to write up a schema for this schema manager which would basically be a .proto file, maybe a GRPC querier, and some table/index definitions. I'm going to try to put this together as a draft PR that we can then look at and try to see if it answers the client workflow questions you have.

[aaronc]

So how about something like this as a starting point:

message StoreSchema {
    string prefix = 1;
    string name = 2;
    oneof schema {
        TableSchema table = 3;
    }
}

message TableSchema {
    string model_type = 1; // the protobuf type name of the model type
    repeated string primary_key_columns = 2;
}

maybe with prefix or name as the primary key of a StoreSchema table. And then there would also be a query endpoint to discover the protobuf schema of the model types. Would that convey enough information about how to decode the store? Then clients could potentially use that for code generation...