[Codec] Implement an Interface Registry

h5law commented 1 year ago

Objective

The Problem

ICS-02 defines numerous interfaces for types that are defined on a per-client basis. For example the ClientState interface will have a different implementation depending on which client it is representing. Each of these implementations must be serialisable as they need to be transferred as []byte types.

This means that we need a way for an interface to be serialised and deserialised depending on the type of client.

Assume the following interface definition:

type ClientMessage interface {
    proto.Message

    ClientType() string
    ValidateBasic() error
}

As proto.Message is embedded in the interface the regular codec.GetCodec().Marshal(interfaceInstance) will work. However, to deserialise the same []byte returned we will need the knowledge of the underlying protobuf definition that implements this interface.

The Solution

In order to enable us to marshal/serialise and unmarshal/deserialise interfaces without the knowledge of their implementations at compile time we must introduce an interface registry. Cosmos uses a similar approach which, we could simplify and alter for our specific use case, see: interface_registry.go.

The general idea is that we utilise anypb.Any{} as well as the protoregistry package, and potentially a new InterfaceRegistry to wrap the two, map type that allows for the following:

Register an interface in the registry
Register implementations of existing interfaces in the registry
Allow for the (un)marshalling / (de)serialisation of interfaces into and out of anypb.Any{} types utilising the protoregistry and InterfaceRegistry wrapper.

This will allow us to serialise interfaces and deserialise them without needing to know exactly which implementation of said interface was originally used. This allows us to take a more general approach to light client definitions, as each client type will define its own implementations of the required interfaces.

Origin Document

IBC Light Clients are required to use specific type definitions which will vary from client to client ICS-02.

We need to be able to marshal and unmarshal these types in order to pass them in and out of the client implementations for verification, as well as for storage reasons in the IBC store & PostgresDB.

// ClientState is an interface that defines the methods required by a clients
// implementation of their own client state object
//
// ClientState is an opaque data structure defined by a client type. It may keep
// arbitrary internal state to track verified roots and past misbehaviours.
type ClientState interface {
    proto.Message

    ClientType() string
    GetLatestHeight() Height
    Validate() error

    // Status returns the status of the client. Only Active clients are allowed
    // to process packets.
    Status(clientStore ProvableStore) ClientStatus

    // GetTimestampAtHeight must return the timestamp for the consensus state
    // associated with the provided height.
    GetTimestampAtHeight(clientStore ProvableStore, height Height) (uint64, error)

    // Initialise is called upon client creation, it allows the client to perform
    // validation on the initial consensus state and set the client state,
    // consensus state and any client-specific metadata necessary for correct
    // light client operation in the provided client store.
    Initialise(clientStore ProvableStore, consensusState ConsensusState) error

    // VerifyMembership is a generic proof verification method which verifies a
    // proof of the existence of a value at a given CommitmentPath at the
    // specified height. The path is expected to be the full CommitmentPath
    VerifyMembership(
        clientStore ProvableStore,
        height Height,
        delayTimePeriod, delayBlockPeriod uint64,
        proof, path, value []byte,
    ) error

    // VerifyNonMembership is a generic proof verification method which verifies
    // the absence of a given CommitmentPath at a specified height. The path is
    // expected to be the full CommitmentPath
    VerifyNonMembership(
        clientStore ProvableStore,
        height Height,
        delayTimePeriod, delayBlockPeriod uint64,
        proof, path []byte,
    ) error

    // VerifyClientMessage verifies a ClientMessage. A ClientMessage could be a
    // Header, Misbehaviour, or batch update. It must handle each type of
    // ClientMessage appropriately. Calls to CheckForMisbehaviour, UpdateState,
    // and UpdateStateOnMisbehaviour will assume that the content of the
    // ClientMessage has been verified and can be trusted. An error should be
    // returned if the ClientMessage fails to verify.
    VerifyClientMessage(clientStore ProvableStore, clientMsg ClientMessage) error

    // Checks for evidence of a misbehaviour in Header or Misbehaviour type.
    // It assumes the ClientMessage has already been verified.
    CheckForMisbehaviour(clientStore ProvableStore, clientMsg ClientMessage) bool

    // UpdateStateOnMisbehaviour should perform appropriate state changes on a
    // client state given that misbehaviour has been detected and verified
    UpdateStateOnMisbehaviour(clientStore ProvableStore, clientMsg ClientMessage)

    // UpdateState updates and stores as necessary any associated information
    // for an IBC client, such as the ClientState and corresponding ConsensusState.
    // Upon successful update, a list of consensus heights is returned.
    // It assumes the ClientMessage has already been verified.
    UpdateState(clientStore ProvableStore, clientMsg ClientMessage) []Height
}

// ConsensusState is an interface that defines the methods required by a clients
// implementation of their own consensus state object
//
// ConsensusState is an opaque data structure defined by a client type, used by the
// validity predicate to verify new commits & state roots. Likely the structure will
// contain the last commit produced by the consensus process, including signatures
// and validator set metadata.
type ConsensusState interface {
    proto.Message

    ClientType() string
    GetTimestamp() uint64
    ValidateBasic() error
}

// ClientMessage is an interface that defines the methods required by a clients
// implementation of their own client message object
//
// A ClientMessage is an opaque data structure defined by a client type which
// provides information to update the client. ClientMessages can be submitted
// to an associated client to add new ConsensusState(s) and/or update the
// ClientState. They likely contain a height, a proof, a commitment root, and
// possibly updates to the validity predicate.
type ClientMessage interface {
    proto.Message

    ClientType() string
    ValidateBasic() error
}

// Height is an interface that defines the methods required by a clients
// implementation of their own height object
//
// Heights usually have two components: revision number and revision height.
type Height interface {
    IsZero() bool
    LT(Height) bool
    LTE(Height) bool
    EQ(Height) bool
    GT(Height) bool
    GTE(Height) bool
    Increment() Height
    Decrement() Height
    GetRevisionNumber() uint64
    GetRevisionHeight() uint64
    String() string
}

Goals

Implement an InterfaceRegistry that allows for the marshalling of interfaces and the unmarshalling of []byte back into interfaces
Enable the InterfaceRegistry to be exposed and utilised outside of the codec package
Implement tests that cover the (de)serialisation of interfaces to and from []byte types

Deliverable

[ ] Implement a new InterfaceRegistry type that wraps the anypb.Any{} object and protoregistry package
[ ] Add the ability to register new interfaces, their implementations and names
[ ] Add the ability to serialise interfaces through the InterfaceRegistry
[ ] Add the ability to deserialise interfaces through the InterfaceRegistry
[ ] Add the relevent Pack and Lookup functions required to:
- [ ] convert interfaces into anypb.Any{} objects for serialisation
- [ ] deserialise []byte back into their original implementation types and return the original interface
[ ] Add unit tests that ensure the InterfaceRegistry enables the above

Non-goals / Non-deliverables

Change how the current codec package marshals proto messages
Break any existing marshalling logic

General issue deliverables

[ ] Update the appropriate CHANGELOG(s)
[ ] Update any relevant local/global README(s)
[ ] Update relevant source code tree explanations
[ ] Add or update any relevant or supporting mermaid diagrams

Testing Methodology

[ ] Task specific tests or benchmarks: make ...
[ ] New tests or benchmarks: make ...
[ ] All tests: make test_all
[ ] LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md
[ ] k8s LocalNet: verify a k8s LocalNet is still functioning correctly by following the instructions here

Creator: @h5law Co-Owners: @h5law

Olshansk commented 1 year ago

@h5law I looked over this briefly but will need to think about this deeply (and at length) next week. off the cuff, I definitely want to avoid adding a component as major as an interface_registry but can't say much until I can propose an alternative.

In the meantime, I wanted to ask if simply have protobufs (e.g. like we do in utility/types/proto/message.proto) and adding custom logic to them (e.g. like we do utility/types/message.go) is not sufficient?

I promise I'll dive deeper into this next week (this has been a long one).

h5law commented 1 year ago

In the meantime, I wanted to ask if simply have protobufs (e.g. like we do in utility/types/proto/message.proto) and adding custom logic to them (e.g. like we do utility/types/message.go) is not sufficient?

@Olshansk this is what I am proposing we have the interface and multiple implementations of the interface, however we do not know which implementation is being used at any single time. We need to be able to serialise and deserialise the interface as is without knowledge of which client is being used. In order to work on the interfaces and not have any knowledge of / abstract away the implementation we need something like this. The actual implementation may be able to be simplified.

We could tightly couple ourselves to cosmos' types and this would make it easier but this limits us to expand to NEAR - or to ETH when they work.

The reason we cannot know the implementation type when we deal with these interfaces is that they are defined in the client implementation themselves - for example a tendermint light client will have a different ConsensusState implementation to a near light client. As such we much pass in and out serialised types that are opaque to the client.

Olshansk commented 1 year ago

We discussed this offline during the team's weekly sync and are looking into potentially using a protobuf oneof as an alternate solution to this.

Whatever approach we decide to go with, let's use an ADR to document tradeoffs: https://github.com/pokt-network/pocket-network-protocol/tree/main/ADRs

How do we serialize define it?

// General
type ClientState interface {}

// Specific
type NearIBCInterace interface {}

// Specific
type SolanaIBCInterace interface {}

// Specific
type PocketIBCInterace interface {}

How do we serialize it?

# Serializble
message ClientState {
    # Include only one (what we need) or none
    oneof client_state {
        // IBC client state for Tendermint
        // Reuse other protos (e.g. tendermint) if we need them
        TendermintClientState tendermint_client_state = 1;
        // IBC client state for Ethereum
        EthereumClientState ethereum_client_state = 2;
        // IBC client state for Solana
        SolanaClientState solana_client_state = 3;
        // intead of registr
        // add an element if we need (nothing else)
    }
}

How do we use it?

switch ClientState.client_state.(type) {
case Tendermint:
    CustomTendermintBusinessLogic()
case Ethereum:
    CustomEthereumBusinessLogic()
case Solana:
    CustomSolanaBusinessLogic()
}

What are our requirments?

Requirements:
1. Serializable & Deserializable (i.e. conversion to / from bytes)
    - E.g. Protobuf, amino, thrift, etc... (protobuf is the most popular)
2. General purpose interface w/o an implementation
    - Golang interface that maps to implementations
3. Where are the implemtantions?
    - A registry
    - Protobufs?
4. What do other others do?
    - Cosmos: registry
    - Near: ???
    - Solona: ???
5. How do we implement?
    - Less code
    - Less maintenance
    - Simpler
    - Easier to understand & test

h5law commented 1 year ago

Closing due to a more appropriate solution being found

pokt-network / pocket