search

Hydrospheredata / hydro-serving

MLOps Platform
http://docs.hydrosphere.io
Apache License 2.0
271 stars 41 forks source link

when accessing a deployment in k8 through the Sidecar no LoadBalancing #263

Closed mynap closed 5 years ago

mynap commented 5 years ago

I have a deployment - say - mv1 in a K8 cluster and I have manually scaled it to 10 pods with a simple Application to ...../gateway/application/add_application

That adds 2 numbers.

When I use the sidecar - and use Kubetail - I only see logs in one of the pods - the rest of the pods are not being used.

..../gateway/application/add_application

If I create a separate service for the deployment mv1 as a LoadBalancer - then 

What should the URL be when accessing the deployment directly?

Please use the above URL snippet as an example.

Thanks

KineticCookie commented 5 years ago

Hi. Let me clarify something.

  1. k8s Deployment entity doesn't provide ability to serve requests. It's just an advanced ReplicaSet. Instead, Service entity manages all the networking capabilities.
  2. You don't need LoadBalancer Service, since all the routing is done INSIDE the k8s cluster. In addition to that, there is no availability yet to add LB address to our model routing table, since it's external for the cluster. So, you need to leave ClusterIP type as-is.
  3. Service uses random/round-robin to route requests. Request routing depends on kubeproxy component and it's configuration.
  4. GRPC (and HTTP/2 I believe), which is used internally in Hydrosphere, aggressively reuses old connections, so there might be inconsistencies between distribution of requests.

On the whole, there is no simple answer nor solution, since it depends on your k8s configuration.

P.S. I found this user group where people discussed similar issues https://groups.google.com/forum/#!topic/kubernetes-users/lvfyKzUf-Vg

KineticCookie commented 5 years ago

And of course an official k8s doc about load balancing: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies

mynap commented 5 years ago

Thanks for the insight and response. Will go through the resources you have shared.

Going back to point 3 of the earlier post - looks like that is the area to focus on - to solve our issue.

Which is after manually scaling the pods that - has a model/application created using Hydrosphere - using the Service interface is the way to go.

Will do my research and get back,

Thanks

mynap commented 5 years ago

After some deep diving into K8 - and using point 3 as the cue - figured out the exposing of a LoadBalancer for a service/.. and now the models that need scaling - scales nicely using HPA!

And we are using gRPC on the client side from an external Akka cluster - smooth ..

Great job with Hydroserving!

Thanks for the suggestions @KineticCookie