search

kubernetes-retired / kubefed

Kubernetes Cluster Federation
Apache License 2.0
2.5k stars 529 forks source link

Add Istio Support #318

Closed danehans closed 5 years ago

danehans commented 6 years ago

Kubernetes is the predominate environment for running Istio. Managing multiple Istio service meshes pose similar challenges as managing multiple Kubernetes clusters.

danehans commented 6 years ago

/cc @marun @pmorie @shashidharatd

gyliu513 commented 6 years ago

@danehans I would love to help on this as well. But have some questions for this:

1) What is the direction for multicluster support with Istio in Federation V2? I can see there are two directions now, one is a single mesh across multiple clusters and another is single mesh per cluster and leverage ingress gateway/egress gateway for across cluster communication.

2) How about open a google doc and propose some design and solution there? We can probably discuss this in weekly meeting to get more comments.

gyliu513 commented 6 years ago

FYI @kevin-wangzefeng @quinton-hoole-2 @irfanurrehman

danehans commented 6 years ago

@gyliu513 it's great to hear that this topic interests you.

What is the direction for multicluster support with Istio in Federation V2? I can see there are two directions now, one is a single mesh across multiple clusters and another is single mesh per cluster and leverage ingress gateway/egress gateway for across cluster communication.

IMO, there is no consensus for addressing multi-cluster within the Istio community. Most work products address extending a single Istio control plane across multiple clusters/environments. See slide 4 for more details. My driver for federating Istio is to treat each mesh/cluster as an independent, yet federated, entity. I have created a gist that describes the Istio federation use cases in more detail. At a high-level, the primary use case for federating Istio is as follows:

  1. I need to run an application that consists of multiple services.
  2. I need to have multiple copies of this app for HA/scalability purposes.
  3. Each copy of the app needs to run in a separate cluster/mesh.
  4. Although multiple copies of the app exist, users connect to a single endpoint (i.e. www.example.com).

How about open a google doc and propose some design and solution there? We can probably discuss this in weekly meeting to get more comments.

Before putting the time into a design doc, I would first like to get consensus from the fed-v2 project maintainers that federating Istio is within the project's scope. If so, I'm happy to document the details in a design doc.

quinton-hoole commented 6 years ago

@danehans Yes, that's definitely in scope. I think tackling 3 first is useful, and relatively low-hanging fruit. I'd really love to expand that to 4 (single public entry point, multiple clusters), and ultimately beyond (multiple clusters in a mesh).

I'm really looking forward to pursuing this further.

quinton-hoole commented 6 years ago

PS: I believe that what you want to achieve in your first phase should already mostly work (pushing copies of your app and istio mesh into multiple clusters, with per-cluster customizations). And we have some stuff to help you with the single endpoint part too.

gyliu513 commented 6 years ago

Thanks @quinton-hoole-2 and @danehans , very helpful information.

Istio community also has some plan to make the multicluster support more stable in 1.x release and this proposal making Istio multicluster leverage some features from federation v2 is a good integration point for those technologies.

@danehans my only concern for this is federation v2 is still under 0.0.2, not sure if it is a good time to do the integration with Istio, but I think it is good to do some prototype or investigation for this based on federation v2. If we cannot get enough feedback from other maintainers, we can probably make this as a topic for next wg meeting.

quinton-hoole commented 6 years ago

Lets go for it I'd say, with the understanding that Federation v2 is still alpha.

danehans commented 6 years ago

@quinton-hoole thank for the feedback. It's great to hear that this issue is considered in scope.

PS: I believe that what you want to achieve in your first phase should already mostly work (pushing copies of your app and istio mesh into multiple clusters, with per-cluster customizations).

"Mostly" work is correct. My initial obstacle in adding Istio support is with propagatedVersion. This type is namespace-scoped and therefore does not support federating cluster-scoped resources such as ClusterRole required for managing Istio. My understanding is that @marun is working on this issue. I have a branch that supports cluster-scoped propagation and passes create/delete CRUD tests, but not update tests. Adding cluster-scoped version propagation will be a big step forward for Istio support.

And we have some stuff to help you with the single endpoint part too.

If you're referring to multi-cluster Ingress/Service DNS, I am familiar with these features and currently in the process of submitting a PR to document Ingress DNS. As I mentioned above, Istio uses the VirtualGateway type instead of Ingress to expose mesh services outside the cluster. We can add add a virtual gateway dns controller and the associated VirtualGateway type, but according to https://github.com/kubernetes-sigs/federation-v2/issues/60, a CRD-based approach is preferred for extending fed-v2. We may need this work to be completed before adding multi-cluster VirtualGateway DNS support.

quinton-hoole commented 6 years ago

@danehans Cool! Sounds like you're on the right track. I was not even aware of the VirtualGateway type in kubernetes. If it's not a standard kubernetes type (including CRD), then you're right, we have no plan to support it, and CRD would be the best route to follow. I assume that would be consistent with your overall direction anyway, as CRD's are the preferred way (and increasingly the only well-supported way) of extending the kubernetes API, to the best of my knowledge.

danehans commented 6 years ago

I referenced VirtualGateway, but I meant Gateway. Gateway is a CRD used by Istio and created before running the Istio control-plane. You can see the CRD details here and learn more about the Gateway type here.

quinton-hoole commented 6 years ago

OK, Gateway should federate fine then.

Q

On Fri, Oct 12, 2018 at 11:51 AM Daneyon Hansen notifications@github.com wrote:

I referenced VirtualGateway, but I meant Gateway. Gateway is a CRD used by Istio and created https://istio.io/docs/setup/kubernetes/quick-start/#installation-steps before running the Istio control-plane. You can see the CRD details here https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/templates/crds.yaml#L67-L86 and learn more about the Gateway type here https://istio.io/docs/reference/config/istio.networking.v1alpha3/#Gateway .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/federation-v2/issues/318#issuecomment-429424494, or mute the thread https://github.com/notifications/unsubscribe-auth/ApoAeEuttLyAGK-u16Q9dGX6MuYGbnvDks5ukOSmgaJpZM4XWO4m .

-- Quinton Hoole quinton@hoole.biz

danehans commented 6 years ago

Yes, after fed-v2 supports cluster-scoped propagated version. @gyliu513, maybe you can take a look at my branch and provide some thoughts on why it's not passing CRUD "Update" tests. This is a blocker for me proceeding with the Istio support effort.

gyliu513 commented 6 years ago

@danehans are you encountering this issue https://github.com/kubernetes-sigs/federation-v2/issues/214 ?

danehans commented 6 years ago

I have a working implementation of Federated Istio with the details documented here. @marun felt it's best for the Federated Istio manifests, documentation, etc. to reside in the Istio community instead of the federation-v2 example directory, so I created https://github.com/istio/istio.io/issues/2843 to address the logistics.

glerchundi commented 5 years ago

Is it possible to made intra-federated-cluster communication? I.e. connect from a ServiceA located in Cluster A (Taiwan) to ServiceB located in ClusterB (Spain)? Imagine a compliance restriction that forces a microservice to stay in a concrete region/cluster (but allowing in-transit communications) how would you allow communication for a potential user in another region can access to that service?

/cc @quinton-hoole @quinton-hoole-2

quinton-hoole commented 5 years ago

@glerchundi Yes, yes, that's cross-cluster service discovery. @shashidharatd implemented it, and can give you the details. It's currently implemented based on DNS and public load-balanced services. Istio support would further extend this by allowing access to non-public services between clusters.

glerchundi commented 5 years ago

Fantastic, thanks @quinton-hoole-2!

marun commented 5 years ago

Given that this work is no longer actionable for this repo (beyond the already-implemented support for propagation of cluster-scoped types), I'm closing this issue. Feel free to reopen.