Events

[iasoon]

Currently, we have a bit of a non-explicit event-driven architecture:

• The gameserver sends out messages ('events') to clients, which they respond to.
• The game implementation explicitly acts on an event channel, which includes client responses.

Now, we wanted to add logging to the system. For the scope of this issue, we are interested in match logs, which record all events (there we have that word again) that occur within one match, so that it can be properly visualized, analysed, and debugged. A really neat way of achieving this would be make 'events' a first-class concept in MOZAIC. The gameserver would then send out events to clients. Each client would log the event stream they receive, and responds to the appropriate events (such as a prompt for a command). Remember that we wanted to do gameserver logging through a client as well, by sending it all log messages. That would be trivial in this scheme, the 'logger' would just be a client that never responds to events.

A scheme like this would explicitly log all communication (thus, all client and gameserver input) which means that it could also be replayed, without requiring any effort on the game implementer's side.

Now, while the basic idea is quite simple, there are a few details we should take into account.

Firstly, we would probably want to merge the message/response scheme we have now with this event scheme. If we wouldn't do this, every event would be a 'message sent' event, which is quite useless. Merging these systems would also provide each event with an uuid, which could come in handy in the future. We'd have to replace the concept of a 'response message' with a 'responds to' tag which would be attached to a regular message (because the response also needs an id in this scheme). This quite okay, and would also enable us to do three-way exchanges instead of just two-way.

Secondly, it is not clear to me how exactly events should flow. Sending from the gameserver is quite straightforward: the gameserver assembles events and dispatches them to the appropriate client. On the client, we are facing a different story. The easy part is the messages received from the gameserver, which we can just log. Then, we would have to send responses. I guess these responses would also be written to the client log file. Of course, we would need a strict distinction between received and produced events, because one is input and one is output. Should these messages also be logged on the gameserver (so, on the logging client belonging to the gameserver)? How should this happen? Do we wrap the received events in an 'event received' enveloppe? Thirdly, there are events that should never be sent over the wire. An example would be error output produced by a bot. This output should only logged locally by the client, and never shared with the server.

An additional idea we might want to take into account: building further on the idea of clients as event handlers, we could do this in a much more fine-grained way. For example, if a bot runner emits events when its bot produces output or an error, the client would also handle this 'message stream' in addition to the one it receives from the gameserver. This is neat because it provides a very traceable stream of data, while also allowing us to stub out that stream for a recorded stream. This could be invaluable for testing. It could also provide a very clear and testable interface for component implementers.

So, let me try to summarize the requirements:

• Each event should have an identifier unique to the match it happened in.
• We want to be able to know where an event originated (eg locally in a client, or on the gameserver). Note that events only travel between a client and the gameserver, never from client to client.
• We need a way to decide which events to propagate from the client to the gameserver. Is this the same distinction as between local and non-local events?

The question now is to design a format that addresses all these concerns, which we can use for the actual logging. And whether I identified all different cases that we have to account for.

[wschella]

I'd rephrase 'error output produced by bot, ... handled by server locally ...'. I have some other comments i'll type up when I have a pc.

[iasoon]

About the integration of the messaging system with this event system (for a recap on what and why that is, please refer to https://github.com/ZeusWPI/MOZAIC/issues/156#issuecomment-383290919 ).

I feel quite confident that we can merge the 'Message' type with events. So, instead of sending a 'message', we are sending an 'event', that is all. These events will have a match-unique identifier given by (client_id, message_number), where message_number is taken from the current message system. An event could look like this:

Event {
  id: int64
  type_id: int64
  content: bytes
}

id is thus a made up of the client and message number, as described above. type_id is a number that identifies the message type, and thus defines the content format. For example, it could be, say, 150, which could identify a client_connected event type.

For the response part, we have to main options: either we integrate it into the event system, or we wrap it. The first option could look like this:

Event {
  id: int64,
  type_id: int64,
  content: bytes,
  optional responds_to: int64,
}

where responds_to is an optional field that would refer to the event id that this event responds to. For the second option, we don't change the event format, but rather we reserve a type_id for responses. A response event could then look like this:

Event {
  id: int64,
  type_id: SOME_CONSTANT,
  content: {
    responds_to: int64,
    content: bytes,
  }
}

The main difference is that in the first case, the response has a type_id, and in the second it does not. I'm not sure which is better. The first case allows for easier parsing by the framework, while the second offers more liberty for the game implementer, I guess.

[iasoon]

WAIT

An additional idea we might want to take into account: building further on the idea of clients as event handlers, we could do this in a much more fine-grained way. For example, if a bot runner emits events when its bot produces output or an error, the client would also handle this 'message stream' in addition to the one it receives from the gameserver. This is neat because it provides a very traceable stream of data, while also allowing us to stub out that stream for a recorded stream.

The missing part of the puzzle is that we have to apply the same idea to client-side logging as to the server-side logging: the log comes first, and all things that actually happen are reactions to the log.

Event {
  id: int64,
  type_id: SOME_CONSTANT,
  content: {
    responds_to: int64,
    content: bytes,
  }
}

When we send a message to the gameserver from the client, we want to log that. Now, flip it around: instead of having the sending of the message trigger the logging, have the logging trigger the sending. This way, the entire client implementation works through computational effects (in short, instead of executing side effects, you describe them and an "abstract machine" executes them. The 'description' here is the log entry). You would program a client through setting 'triggers' on certain events (callbacks, handlers, whatever you call them). The client library would be no different, and have callbacks on a set of library events.

MY DUDES

[wschella]

Some unorganized personal opinions/reflections/comments:

• Events are great for async! Not always for ordering things tho. Specifically, the order in which events are processed will be important, but networks kinda make that hard (as you personally have experienced trough building MOZAIC). This is just a cautionary warning to keep in mind tho, as we have this difficulty already. Question arises: in which layer do we want to guarantee ordering, do we even want to do it ourselves? (id is quite helpful already)
• Is type_id something predefined by MOZAIC? Are they globally unique (over matches)?
• I don't really understand the 'every message is a send event', but I agree on merging the Req/Res and event system.
• Is 'message received' an event? Both for client and server. It is right?
• As you said, we need a distinction between message received and send, but It would be nice if they were kept in the same place.
• I liked the optional responds_to more, as to avoid nesting of things. But that makes parsing a lot more difficult. However, this does too. Just not sure about how we actually want to do it.
• I really like merging server and bot event streams. We should be able to make the distinction tho.

Also some concern around the 'log comes first, it triggers the sending' way of wording it. The log would just be a homomorphic clone of the event stream. Computational effects are defined on the event stream, and logging is one them.

[iasoon]

Thank you for your comments @wschella! I'll try to address your questions as precise as I can.

On ordering

I don't think ordering will be a real issue. There are two main event streams that arrive in the client: events that are sent from the game server, and events that are produced locally. Events produced by the game server should be ordered by the game server itself if so desired, and the networking layer will maintain this ordering. Assuming the client processes events synchronously, its event stream is also ordered. So, issues can only occur in the merging of these streams, but since the game server stream suffers from networking delays, we cannot make ordering guarantees here anyways. When necessary, the game implementation should avoid re-ordering issues itself, for example by using the synchronization primitives we offer (like request/reply).

On type ids

I think type ids should be fixed, so that it is possible to decode an event without context. We'd certainly have a set of predefined framework types (such as network events), and we'd also certainly have user-defined types (such as anything that happens in a game).

A nice way to go about this would be to assign 'hierarchical ranges' - similar to how HTTP status codes and IP addresses are assigned - to the different components that make up the system. For example, 0-255 could be reserved for framework events, and 256-511 for user events.

Of course, different games would have different event types. I feel the most ergonomic compromise would be to not expect type ids to be MOZAIC-unique, just game-unique (so planetwars and higher-lower can both assign message type 420 as they please). While this introduces a little context in decoding (one can not just parse any MOZAIC log), I suspect the alternatives will introduce a little headache. Most importantly, I think this scheme is easy for a game implementer to wrap his head around.

On events and messages

It seems like I wasn't really clear about my intentions here. Since we can only log on clients, the game server can never log an event, only tell a client about it. Since sending data to a client is exactly telling a client about something, we can just drop the distinction between sending data and logging; they are both messages being sent to a client. This makes sense because we want to log all received communication anyways. The difference between what previously was a 'log' message or a 'game' message is then only made by the processing of this event: the game message will most likely result in some action being taken by a client (such as propagating the game state to the bot it is running). The real nice thing here is that this also makes it trivial to register triggers on what would otherwise just be a 'log' message, such as a client connecting (which is a networking event and would thus get logged by the framework. Logging now means 'sending the event to the game owner', who can then react to it by, say, showing it in some UI). So, to answer some questions: 'message received' is not an event on a client, because the event is the 'message received' event. It is an event on the server, because the server cannot log the input it received, but has to tell a client about it (which it can only do by dispatching an event).

I am not 100% sure of this part yet, but I think we want to model the entire system as 'events that flow towards clients' This means that every event will have a 'client_id' annotation (which is the client which the event was sent to). So, all data sent from the server to the client is an event in this scheme, as described above. Data sent from the client to the server would then obviously not be an event, since it wasn't sent to a client. Except when we still make it an event, on the client that sent the data. This is where the computational effects come into play: by declaring that we are sending data, it will actually be sent, and recorded. This works by just registering a callback on the 'send_data' event, which will execute the effect. When the game server received the data, it will emit an event to the game owner client that it received some data. With the omission of event directionality, we no longer need a distinction between messages sent and received. Side note: responses would then be implemented with a custom 'send_response' event for responses being sent by the client, and a 'response' event for responses sent by the game server. (Or if we decide to go with the other scheme, apply an analogous transformation).

About the origin of events: while this is not required for any functionality (as far as I can see), it would indeed be very convenient, and also necessary to avoid message id collisions (since ids will both be assigned by the game server and the client itself). We can then have unique ids by using origin_id#event_number. Perhaps it would be nice to apply a scheme similar to the hierarchical event types to these origin ids, allowing us to specify the exact component that sent the event (such as the game server networking module, the planetwars implementation, the bot runner, ...). This could also prove useful for log filtering et cetera. I'm not sure whether I like this idea, suggestions would certainly be welcome.

About your final remark: You are right, I meant the event stream and not the on-disk log. But since you're nitpicking: I think the second part is wrong, Computational effects are not defined on the log stream, they are described by the log stream. Logging is not one of them, since the event stream does not describe that logging happens. Logging is rather just an 'unrelated' capture of the stream.

[iasoon]

Implemented on the REACTORS branch.