Protocols as generators

[dbluhm]

This PR introduces a pattern of programming protocols as async python generators, including some helper functions to run those protocols and retrieve relevant information yielded by the generator.

I feel this is a fairly succinct way of creating flexible "protocols" that can be interrupted and resumed or only run to a certain point while keeping the method that implements the protocol clean and readable. That being said, I'm open to thoughts and feedback on the approach.

Taking the inviter role of the connection protocol, we get the following "protocol generator":

async def _inviter(config, temporary_channel):
    """Inviter protocol generator."""
    invite = Invite.parse_invite(
        input('Input generated connection invite: ')
    )

    yield 'invite', invite

    # Create my information for connection
    invite_key, invite_endpoint = invite.get_connection_info()
    with temporary_channel(invite_key, invite_endpoint) as conn:

        yield 'before_request', conn

        request = Request.make(
            'test-connection-started-by-tested-agent',
            conn.did,
            conn.my_vk_b58,
            config['endpoint']
        )

        yield 'request', conn, request

        response = Response(await conn.send_and_await_reply_async(
            request,
            condition=lambda msg: msg.type == Response.TYPE,
            timeout=30
        ))

        yield 'response', conn, response

        response.validate_pre_sig_verify()
        response.verify_sig(invite_key)
        response.validate_post_sig_verify()

        yield 'response_verified', conn, response

        _did, conn.their_vk_b58, conn.endpoint = response.get_connection_info()
        conn.their_vk = crypto.b58_to_bytes(conn.their_vk_b58)

        yield 'complete', conn

Each yield represents a point at which the protocol can be stopped or values collected.

To simply run the protocol to completion:

inviter = _inviter(config, temporary) # Create the generator; this will normally be a fixture
await run(inviter)

To run the protocol but stop it at a certain point:

await run(interrupt(inviter, on='response_verified'))

Or to return the yielded values at that point:

conn, response = await last(interrupt(inviter, on='response_verified'))

To resume the protocol:

inviter = _inviter(config, temporary)
conn, request = await last(interrupt(inviter, on='request'))
# Do something else, maybe modify request
await run(inviter) # Resume the protocol

To collect all messages as a list:

messages = await collect_messages(inviter)

To collect all messages as a list to a point in the protocol:

messages_through_response = await collect_messages(interrupt(inviter, on='response'))

To get a map of messages emitted where the keys of the map are the "events":

message_map = await event_message_map(inviter)

(which can also be interrupted, yielding a map with all messages up to a point)

And so on.

Again, I'm open to thoughts and feedback. I think this could be a useful structure but certainly wouldn't be a requirement for all protocol tests.

[dbluhm]

If you look at pytest fixtures, you will see that they can be written as generators, too. See https://pytest.readthedocs.io/en/2.9.1/yieldfixture.html. What you yield is the starting state provided by the fixture. But after yield is called inside the fixture to provide that state, pytest also runs the function to completion, and the completion code does the cleanup/teardown. This might be a nice way for protocol generators to work, because it would mean my cleanup/teardown code for the protocol can be located next to the code that creates the state.

Did some experimentation. We can't use the protocol generator directly as a fixture since, after the first yield, the rest of the method would be treated as cleanup and we wouldn't be able to use anything after the first yield in test functions that pull in that fixture.

We could wrap the protocol generator in a fixture, i.e.

@pytest.fixture
async def inviter(config, temporary_channel):
    async def _inviter():
        #inviter stuff

    generator = _inviter()
    yield generator
    await generator.aclose()
    # Clean up

That feels a little odd to me though.

An alternative for keeping teardown next to our state generating code might be to wrap the contents of the generator in a try/finally:

async def _inviter(config, temporary_channel):
    try:
       #inviter stuff
    finally:
       # clean up

@pytest.fixture
async def inviter(config, temporary_channel):
    generator = _inviter(config, temporary_channel)
    yield generator
    await generator.aclose()

In this setup, the clean up will occur after completely consuming the generator or after the test function executes on generator.aclose(), triggered by fixture clean up.

The differences between these two experiments are somewhat marginal anyways though lol.

I'm not positive I grasped your intent with your suggested enhancement; let me know if I missed your point.

[dhh1128]

I can see how I've been unclear. Sorry about that. I wasn't recommending that we actually use pytest fixtures to build protocol generators. I was suggesting that we could handle protocol generators in the same way that pytest handles fixtures, since that would give us a clean way to tear down protocols, not just set them up.

This isn't a big ask from me--just an interesting idea I wanted to share. So feel free to plunge ahead without this. Maybe just bookmark the idea.