Send Message history in (Human)AgentBrain

[thaije]

Is your feature request related to a problem? Please describe. In the MATRX visualizer, clicking on one of the context menu options sends a message. However, request for sending a message is sent to the API which adds it directly to the messages to send in MATRX. Because this is done directly, it is not possible in the HumanAgentBrain to see what messages were actually send directly from the frontend to another agent, as the message is never put in self.messages_to_send in the humanagent.

Describe the solution you would like As such, the request for a new (human) agent property: SendMessagesHistory. This should include all messages send by the current human agent, including directly from the API. E.g. per tick.

This also matches with the idea that received_messages are not removed each tick by default (issue #112), but it is up to the user to remove those or instead use a "read" message property, and only reading new ones. Effectively also being a received_messages_history.

Describe alternatives you have considered Put API messages in agent.messages_to_send. However, this doesn't work, as the API messages are appended and popped from there after the agent their OODA loop.

Additional context In Co-active teaming we wanted to visualize any open requests in the frontend. However, as these requests were send via the context menu we couldn't filter the objects in filter_observations and change their visualization.

[jwaa]

So if I understand correctly, the message system is used for the context menus but this circumvents the current implementation of self.messages_to_send (at the sender's side) and self.received_messages (at the receiver's side)? This prevents a human agent brain from being "aware" what messages were send in its name by a human.

Why do you need this "awareness"? I can imagine that you might want to store this with a Logger, and the obvious point of access would be the AgentBrain.log() method for that. However, I can't figure out a different reason.

It is important that MATRX is capable of logging all interaction a human has with a GridWorld. Given the new context menu, I believe this reason alone would warrant this functionality. But I am curious if there are other reasons.

In the past we discussed additional improvements of the message system, so let's see if we can bundle those ideas and features such as these into a single overhaul. The MessageHandler is a good startingpoint here, but I think there is still more to gain. This raises the question; when do we plan this message overhaul? Is it the major improvement of v1.2 (favoured over the simplification of creating agents or the event system)? Or do we want to postpone this to a later version v1.x?

P.s. I will make a stub issue for the potential message system overhaul where we can discuss that topic in more detail.

[thaije]

@jwaa as it is implemented now there is no issue on the MATRX core side of things. The context menu sends a message, which is put in the gridworld.message_manager. That is available from all loggers, so there is no issue there.

The issue is that sent messages on behalf of a HumanAgent using a context menu are not in any way accessible in the HumanAgentBrain. Ofcourse, they end up in the self.received_messages, but only if the receiver agent is you (which might be you, but might also be someone else if you are tasking another agent).

The concrete example for why this is an issue: You are controlling a human agent. In the frontend with a context menu, you sent a message to another agent, requesting that other agent to do a specific task (=an object). Now in the frontend you want to display this for the user, by drawing a circle around that object. However, that is not possible because you can never see the request/message from the HumanAgentBrain you yourself send in the to another agent via the context menu. As the context menu messages are not put in self.messages_to_send.

[jwaa]

@thaije Good point, I forgot that the MessageHandler lives in the GridWorld, and is thus accessible by all "world loggers".

You provide a concrete example of how a HumanAgentBrain wants to change something based on the command a human user sends to another agent through a context menu. Is it also possible that this change, besides being visual, is also actually in the behaviour of the HumanAgentBrain itself? For example, when the human sends a custom message ("go left, I go right") through the context menu to another agent, this also includes behaviour for that human agent itself (the selected agent goes left, the human agent goes right)? Ofcourse it would be nice if this would be possible, but you might achieve the same with the agent sending a reply back to the HumanAgentBrain.

I ask these questions just to make sure if it is something we need and if so (and it seems it would be quite handy) when we need it. So we can plan it accordingly, as our next milestone is filling up quite nicely we need to restrain ourself a bit :)