search

andydunstall / piko

An open-source alternative to Ngrok, designed to serve production traffic and be simple to host (particularly on Kubernetes)
MIT License
1.73k stars 36 forks source link

Upstream load balancing #30

Open andydunstall opened 1 month ago

andydunstall commented 1 month ago

Say you have 10 upstream connections for endpoint E. Piko should attempt to evenly distribute of load among those upstream connections.

If node N has upstream connections to its local node, it will distribute connections among those upstream connections in a round-robin fashion.

However if N doesn't have an upstream connection itself, so must forward to another node, it should still attempt to load balance requests evenly among the known upstream connections. Such as if node N1 handles requests for endpoint E, and it knows node N2 has 10 upstream connections for the endpoint, and N3 has 2 upstream connections for the endpoint, then it should send 5 times more requests to N2 than N3 (those nodes with then load balance among their connected upstreams).

dmattia commented 1 month ago

How does this work right now if there are multiple upstream connections for endpoint E?

andydunstall commented 1 month ago

If node N has upstream connections to its local node, it will distribute connections among those upstream connections in a round-robin fashion.

This is local load balancing is already implemented for v0.1.0

However if N1 doesn't have a connected upstream, it simply iterates through its list of known nodes (in insertion order) for the first node that is advertising as having a connection for that endpoint. So N1 could end up always forwarding requests to the same node for endpoint E even if other nodes have connections

(It was just the quickest option to implement for v0.1.0, but looking will do this properly for v0.2.0)