integrate with kubernetes

[leslie-wang]

It means it will watch on kubernetes service, and add/delete service automatically.

[kobolog]

That's a good idea. Willing to lend a hand and send a PR?

[leslie-wang]

Will do eventually. Right now, got some handy work. Will do once it is done.

[fcantournet]

I'm very interested in doing something like this. Ideally I'd like to run gorb on all my kubernetes nodes, and have only one instance of gorb carrying the service IPs from k8s. (later I expect to be able to remove this limitation and use ECMP routing) I'd like to use this to remplace the kube-proxy, with the added feature (for me) that the service IP is actually routable outside of k8s.

@kobolog I think some of this is in line with the roadmap in the README : What do you think?

[leslie-wang]

@fcantournet kube-proxy performance is bad and can not be used for media application. This is Gorb can help. what I'm thinking is

1. add gorb as daemon set so as to running at each k8s nodes
2. each gorb pod has another container inside to coordinate with the whole gorb cluster, and determine which gorb instance carrying service IP, then invoke gorb API for provisioning
3. develop one k8s add-on to listen and dynamic add/delete new service. Any design in your mind? shall we create one slack team to brainstorm.

[resouer]

@leslie-wang I don't think ipvs NAT mode perform much better than iptables NAT. They are essentially the same thing. Unless you have idea about how to use DR mode with overlay (or other cross-host network)

But one of the benefit I can see is ipvs has less rules on node :)

[leslie-wang]

@resouer that is absolutely correct. I'm only interested in DR mode. Seems like gorb still miss this feature for now. We probably need do it in parallel.

[resouer]

FYI, ref https://github.com/kubernetes/kubernetes/pull/38817

[starsdeep]

@leslie-wang DR mode may be not general for all case, since it requires ipvs director and real server on the same L2 network.

BTW, k8s requires: clients in containers and on host can both access a service through the vip. Adding a bunch of ipvs rules on host works only for clients in containers, because ipvs does not support client and director on the same machine.

I have tried this:

ip link add type dummy
ip addr add <service-ip> dev dummy0
ipvsadm -A -t <service-ip>:<port> -p -s rr
ipvsadm -a -t <service-ip>:<port> -m -r <pod-ip>:<port>
ipvsadm -a -t <service-ip>:<port> -m -r <pod-ip>:<port>

then docker run -ti busybox curl <service-ip>:<port> works, however curl <service-ip>:<port> dose not work, since ipvs does not support client and director on the same machine.

[kobolog]

@leslie-wang have you thought about using IPIP instead of DR? This is the most widely used mode since it offers same direct response benefits, but allows you to cross L2 boundaries.

@starsdeep AFAIK, IPVS in fact does support client and director on the same host and this was actually the way it's used in https://github.com/cocaine/cocaine-core. Let me double check and look into why your example exhibits a different behavior.

[leslie-wang]

our current deployment is mainly on-premise data center. So L2 is ok. IPIP is definitely useful in cloud environment, but has some extra overhead.

[kobolog]

Hey @starsdeep, I've just commented on https://github.com/kubernetes/kubernetes/issues/17470#issuecomment-270428838 re: IPVS and the same-host access – it works.

[murali-reddy]

@leslie-wang @fcantournet Just in case if it interests there is https://github.com/cloudnativelabs/kube-router for Kubernetes that has support for IPVS and runs gobgp so you could advertise cluster IP to external nodes as well.

[szibis]

Maybe this could be solution https://github.com/kubernetes/kubernetes/issues/44063

[kobolog]

I guess I'll close this issue for now since it looks like there's support for IPVS in kube-router for it.