Improve documentation for validating agent responses

[crypdick]

I need my agents to respond in specific formats, and so I want to validate their messages and respond with an error message if they give malformed responses. I see that the Chess example makes use of register_reply to validate the agents make legal Chess moves, but the subclassed AssistantAgent are not well documented and neither are the API docs.

Can someone explain how to validate an agent's outputs?

[victordibia]

Hi @crypdick ,

In general, register_reply can be seen as a function that get's called once the agent generates a response. We can repurpose this for a task such as printing the agents reponse, sending it over a socket etc. or validation in your case.

In the sample below, I have an example of printing the response from the agent

def print_messages(recipient, messages, sender, config): 
    if "callback" in config and  config["callback"] is not None:
        callback = config["callback"]
        callback(sender, recipient, messages[-1])
    print(f"Messages sent to: {recipient.name} | num messages: {len(messages)}")
    return False, None  # required to ensure the agent communication flow continues

user_proxy.register_reply(
    [autogen.Agent, None],
    reply_func=print_messages, 
    config={"callback": None},
)

assistant.register_reply(
    [autogen.Agent, None],
    reply_func=print_messages, 
    config={"callback": None},
) 

You should be able to modify the print_messages function for validation e.g.

• inspect the last message in messages messages[-1]
• perform your validation
• raise some error

It might help to share a toy example of your task that we can use for debuggin purposes.

[crypdick]

Thanks for the quick response @victordibia ! I tried to adapt your code, but it isn't quite working since the error feedback isn't being sent back to the agent.

I've tried to make a minimal example here. The goal is to have a separate validator for different agents, and if a given agent fails a validation, to send a message back to that agent so that it can fix its mistake.

[crypdick]

Hey @victordibia , I wanted to follow up and see if you have had the chance to review my toy example. I haven't figured out how to get the callback working so that I can send error messages back to an agent.

[Nisarg1112]

The goal is to have a separate validator for different agents, and if a given agent fails a validation, to send a message back to that agent so that it can fix its mistake.

@crypdick @victordibia Hey, I'm also looking for similar usecase. Do we've any documentation or example code around this usecase?

[ekzhu]

@Nisarg1112 The chess notebook has a complete example: https://microsoft.github.io/autogen/0.2/docs/notebooks/agentchat_nested_chats_chess/