search

PaddlePaddle / PaddleCloud

PaddlePaddle Docker images and K8s operators for PaddleOCR/Detection developers to use on public/private cloud.
Apache License 2.0
281 stars 75 forks source link

Paddle Cloud need to use host network of kubernetes. #449

Open drinktee opened 6 years ago

drinktee commented 6 years ago

Now paddle run in pods that usually on an overlay network. Overlay may cause some loss of performance of network. Most of the time in deep learning we use a high performance network like RDMA and host network may make sense.

typhoonzero commented 6 years ago

Supporting for the high-performance network is indeed what we needed, but using host network will break the design of Kubernetes.

Kubernetes design the network to be "flat", means containers should treat each other and hosts equally: https://kubernetes.io/docs/concepts/cluster-administration/networking/#kubernetes-model

For RDMA, or for "low-latency" cases, we need to implement this design using hardware accelerator or other technics to achieve this.

drinktee commented 6 years ago

We can use a switch to turn on/off host network. For achieving this there shoud be a component which can figure out available host ports. And this component stand alone without paddle cloud.

typhoonzero commented 6 years ago

@drinktee Yep. Still, I think this solution should be a "workaround", it's not a best technical choice.

drinktee commented 6 years ago

Yes, it's a solution for some special scene.

Yep. Still, I think this solution should be a "workaround", it's not a best technical choice.

tizhou86 commented 6 years ago

We used to submit issue for this, but Kubernetes will not handle host port auto-allocation, so we need to manage the port ourselves. I don't think it's a work around solution, and we need to make it decent.