Refactor `WalletAPI` to match expected reality better

[michaelpj]

Our WalletAPI class currently conflates a number of things:

• Operations on the chain itself
• Operations with smart contracts and their state
• User interactions

I propose we refactor it to make this clearer. We want to match the expected interfaces as much as possible. Here's a suggestion for how to do it to get the discussion started:

class Chain m where
    submit :: SignedTx -> m ()
    -- Or some other suitable query method
    utxo :: m UTxO
    validatePending :: m ()

class WalletBackend m where
    submit :: SignedTx -> m ()
    -- Or some other suitable query method
    utxo :: m UTxO
    -- Could distinguish balanced from unbalanced transactions in types or something
    coinSelect :: Tx -> m Tx

class ContractBackend m where
    -- Handle is anything
    type Handle
    --  State probably a JSON blob to avoid having to have different types per contract
    type State
    -- Output contains Txn to submit and triggers to add to the state. 
    type Output
    init :: Handle -> State -> m ()
    destroy :: Handle -> m ()
    -- Up to the UI to determine what state transformer to pass - we don't want to care about that routing here
    run :: Handle -> (State -> (Ouptut, State)) -> m ()
    -- and either a notify method or a tick method that tells it it might want to poll the backend for changes

-- For writing multi-party test cases, corresponds to `WalletAPI` today
class Driver m where
    type User
    -- Run an action on a particular user's wallet
    userAction :: ContractBackend n => User -> n () -> m ()
    -- Manipulate the chain
    chainAction :: Chain n => n () -> m ()

Here we're mimicking the contract backend as well - this is useful for our tests, which go all the way from user interactions to things happening on the chain. Ideally, this means that the "real" contract backend can be swappable into this picture.

In terms of getting there:

• Pulling out Chain and WalletBackend from WalletAPI should be fairly easy.
• ContractBackend requires rewriting all our contract code to be in state-passing style, but we're going to have to do that anyway. Plus all the tests.

[michaelpj]

This would also makes the testing that Mario's doing easier, since he wants to be able to require less functionality than the whole of WalletAPI.

[j-mueller]

Generally good and the four parts also seem right to me

-- For writing multi-party test cases, corresponds to 'WalletAPI' today

Doesn't it correspond to MonadEmulator?

And am I right in thinking that all the endpoints that currently return a WalletAPI m => m () would be replaced by a State -> (Ouptut, State) functions?

[michaelpj]

I think so - this is based on whatever interface we agree with the app platform folks. We might want to give you a wrapper, so you actually write endpoints with a real state type, and then we just insist that it has to/from JSON instances to be wrapped by the endpoint.

Doesn't it correspond to MonadEmulator?

Perhaps it doesn't correspond exactly to anything we have at the moment. I guess it's close to MonadEmulator since our EmulatorState is set up to model not only the smaller pieces but the whole multi-party situation.

[michaelpj]

I think Mario could easily do at least the first part of this, to avoid further loading Jann and I.

[Sam-Jeston]

@rhyslbw and I spent some time going over this today. We want to propose some changes to the definition of ContractBackend, to better align with the boundaries we've identified within the domain of the "Smart! Contract Backend". Forgive this pseudo-code, I am no Haskell developer! But hopefully the points are self-evident.

--  State probably a JSON blob to avoid having to have different types per contract
type State

-- Output contains Txns 
type Output

-- Events declare state change intention, including the addition of triggers
type Event

-- Script address to target execution
type ScriptAddress

-- Contract is some type that provides the bytecode for execution. Likely what we have been refering to as the "bundle"
type Contract

-- A valid contract method
type Method

-- Serialized arguments for a method
type MethodArgs

class ContractServer m where
    -- Initialize the contract, registering any existing triggers.
    serve :: Contract -> State -> m ()

    -- Execute a specific contract endpoint
    execute :: ScriptAddress -> Method -> MethodArgs -> (Ouptut, Event) -> m ()

class ContractExecution m where
    init :: Contract -> m()
    destroy :: ScriptAddress -> m ()
    execute :: ScriptAddress -> Method -> MethodArgs -> State -> (Output, Event) -> m()

    -- Each contract should expose an apply event endpoint to receive the state changing events to apply
    applyEvent :: ScriptAddress -> State -> Event -> State -> m()

The first distinction is that between the ContractServer and ContractExecution interfaces. The server is called by clients, hence its methods never take state as an argument. Instead, it is responsible for appending state onto the requests and passing them on to the execution engine where the contract is currently exposing its HTTP interface.

We have been using the term execute for calling contract endpoints. But we can use run, call, whatever we agree is most appropriate.

The last difference here is whether or not the execution of contract endpoints returns (Output, State) or (Output, Event). We see a few downsides if state is returned directly, instead of an event that dictates a state change. We lose replay ability if we only ever have reference to current state, and I can just see potential to end up with different state between clients with no chance of being able to derive the correct state between them.

This is likely a future concern, but one I think worth thinking about now. It does mean that contract need to expose an applyEvent handler, specific to events created by that contract, that can apply an event to current state. @rhyslbw may want to expand here.

[michaelpj]

ContractServer vs ContractExecution

So I think there's a point about how we model this in Haskell that I just alluded to that should make this clearer.

In the Haskell side, we have an actual contract endpoint written in state-passing style, say, contribute :: Int -> State -> (Output, State). Now, to use this in your model, of course you don't know the endpoints statically, so you have something doing "routing" to take the endpoint name and the arguments and then pass it all in. But we can just construct the function directly: run (contribute 5) or whatever.

So I think we just don't need ContractServer for our testing.

Returning state vs event

You can always turn a state s into the event state changes to s :) I guess I don't know what you're expecting "an event that represents a state change" to be. Remember these contracts are arbitrary code written by other people. I think getting them to even dump their internal state on every call is going to be difficult, let alone requiring them to have some kind of state update model and return updates.

Life would be easy if we could require that the state of all contracts were CRDTs, but that's not realistic :)

[rhyslbw]

Thanks @michaelpj

Remember these contracts are arbitrary code written by other people. I think getting them to even dump their internal state on every call is going to be difficult, let alone requiring them to have some kind of state update model and return updates.

It's actually a nice style of programming for domain aggregates like smart contracts to explicitly define domain events, express the event has occurred during rule evaluation, and interpret how this event changes the state. It helps develop the contract model, and greatly improves testing capabilities. The concept is to separate the generation of outcomes (events) from the application of what they describe to the state.

For the reliability gains, it simplifies persistence to append-only, supports crash recovery, reduces concurrency issues during network sync, and we gain a data model suitable for push-based APIs and reactive services. Triggers service is an example.

Event Sourcing, Functionality - Arnaud Bailly Haskell + QuickCheck https://www.youtube.com/watch?v=lKIPlARq1z8

[michaelpj]

I just don't see it. It looks like a lot of work for contract authors unless you have magic differentiable programming. Otherwise you can't just manage your state you have to put an extra layer of abstraction in between everything that you do. Or you have to "diff" the output state from the input state before returning.

If the contract execution backend wants to view the outputs of running contract actions as events, I think that's fine, but I don't think we should force that on contract authors. Or if we do they will all give us shitty events that just say "the new state is s".

If you want to record a log of events to synchronize later, why not do it on the other end? I.e. record the events that come in to the contract execution backend (which are presumably going to be comparatively simple since they're coming from the user). Overall I'm pretty sceptical that any non-trivial synchronization is going to work, since there is no guarantee that interleaving two valid streams events will produce a valid stream of events.

Plus, I think explicit state passing gets us most of the benefits you list.

[rhyslbw]

Let's start simple and work with the state as it's returned. Without concrete examples of off-chain state it's all a bit speculative, so we're happy to defer the proposed changes until there's a clearer need during synchronisation.

[m-alvarez]

I've been toying around with these ideas and put together a tentative interface for Chain and WalletBackend that Michael and I think is more or less reasonable, posting here for feedback and discussion.

https://gist.github.com/m-alvarez/970ee35f790b87f8368c070f0488497a

[michaelpj]

Outdated.