Agent Runtime implemented using Ray

[ekzhu]

What feature would you like to be added?

An agent runtime implemented using Ray.

Why is this needed?

Allows for a more robust runtime without leaving the Python ecosystem.

[asheeshgarg]

@ekzhu I see ray actors as a good way to agents implementation. Even I have tested a basic agent with Ray earlier. Would like to contribute more on this. This gives agents restart hosting on GPU/CPU and checkpoint capabilities from the Ray infrastructure.

import ray
from autogen_core.components.models import UserMessage, SystemMessage

@ray.remote
class BaseLLMAgent:
    def __init__(self, model_name, role):
        self.model_name = model_name
        self.role = role
        self.llm = get_azure_chat_openai_model()

    async def generate_text(self, message):
        try:
            messages = [UserMessage(content=message, source="user")]
            response = await self.llm.create(messages=messages)
            return response
        except Exception as e:
            return str(e)

openai_actor = BaseLLMAgent.remote("gpt40",role="You are helpful assistant")

result = ray.get(openai_actor.generate_text.remote("What is the capital of France?"))

print("Generated text:", result)

Generated text: CreateResult(finish_reason='stop', content='The capital of France is Paris.', usage=RequestUsage(prompt_tokens=15, completion_tokens=7), cached=False, logprobs=None)

[ekzhu]

@asheeshgarg can you start a separate repo for this exploration first? We can start a community package for Ray-based runtime for now. Once the design is matured, we can incorporate it as a built-in runtime.

See note: https://microsoft.github.io/autogen/dev/user-guide/extensions-user-guide/index.html

[asheeshgarg]

@ekzhu sure will try that out