Open Future-Outlier opened 2 months ago
Flyte already supports round-robin load balancing for agent server deployments. According to the Agent Metadata Servicer PR, you can configure multiple agents with round-robin load balancing by specifying the defaultServiceConfig
in the agent configuration.
Example configuration:
plugins:
agent-service:
defaultAgent:
endpoint: "dns:///localhost:8000"
insecure: true
timeouts:
GetTask: 100s
defaultTimeout: 100s
agents:
custom_agent:
endpoint: "dns:///localhost:8001"
insecure: true
defaultServiceConfig: '{"loadBalancingConfig": [{"round_robin":{}}]}'
timeouts:
DoTask: 300s
GetTask: 100s
defaultTimeout: 300s
agentForTaskTypes:
- custom_task: custom_agent
- api_task: custom_agent
Would you like more details on how to set this up?
Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @runllm to ask follow-up questions. See runllm.com for more!
Hey @Future-Outlier. I'm not entirely sure if I'm following this. If one endpoint points to a DNS which solves to multiple Ips (if SRV even ports and everything) , won't they be loadbalacing between them?
Ideally I would like to not even think about the port or at least not have a LB based on port since multiple instances of my agent will be all the same port.
I agree with @RRap0so I do not think we should implement a custom loadbalancer. this is a load balancers job.
Motivation: Why do you think this is important?
It can improve performance when we have more than 1 agent deployments and support same task types.
Goal: What should the final outcome look like, ideally?
For example, 2 endpoint (
localhost:8000
andlocalhost:8001
) task typea
1st time send request tolocalhost:8000
, and second time send request tolocalhost:8001
.Describe alternatives you've considered
Stay the same behavior.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?