Flagger support for Consul

oleksiyp commented 4 years ago

Feature Description

I would like to install one helm chart in several DCs and get Consul Connect service mesh out of the box.

Use Case(s)

multi-DC failover
canary deployments

My experience

I spent a few weekends to make this work on PoC level. And here is my experience.

Problems I faced:

standard helm chart doesn't support multi-DC scenario
there are no tools for canary deployments
not clear procedure to create common trust domain
multi-DC failover doesn't work without health-check
exposing health-check in Kubernetes doesn't work
standard statistics exposed through statsd is limited

Didn't find in docs:

how to generate SPIFFE SVID (thus used Vault, although it might be overkill for some cases)
mention that ClusterID and thus trust domain should be same between all DCs

Modified:

consul-k8s project to expose HTTP health checks
consul-helm project to make join WAN possible
- distribution of servers and clients by corresponding node labels
- expose hostPorts
flagger to support Consul Connect Canary deployment and multi-DC failover

I added to service chart additional envoy instance to expose statistics to Prometheus directly through envoys /metrics (although now I realize that I potentially can just expose this path)

Created test-bed:

simple ingress forwarding traffic from outside: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/ingress-chart
Kubernetes controller called prefixrouter to forward traffic to services based on CRD definitions: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/prefixrouter-chart
Service instance which randomly picks up next service-to-service call and does a fixed amount of hops: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/service-ktor
UI client that shows how traffic is distributed: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/ingress-js

As a result:

I joined Consul Connect DCs in two Kubernetes clusters(kind)
I exposed health-check and have an ability to create liveness/readiness probe
I am forwarding traffic through mesh-gateways by sharing ClusterID
I am able to perform Flagger deployments and shift traffic from old version to new version
based on exposed health-checks traffic is redirected to other DC in case of failure

All setup available here: https://github.com/oleksiyp/consul-connect-multi-dc

oleksiyp commented 4 years ago

@nicholasjackson here is full context of my changes related to https://github.com/weaveworks/flagger/pull/482

jsosulska commented 4 years ago

This relates to #7412 . Cross linking for feature request.

david-yu commented 4 years ago

Hi @oleksiyp

Thank you for your feedback. We just released WAN Federation over Mesh Gateways for Kubernetes to enable the multi-dc workflow: https://www.consul.io/docs/k8s/installation/multi-cluster/overview

In addition we are working towards enabling our newly built Ingress Gateway for Consul 1.8 beta (https://www.consul.io/docs/connect/ingress_gateway) into the Helm chart for deployment as well.

As far as Flagger, we have seen requests for flagger in the past. We'll take a review your PR as well so thank you for the PR.

ssimard24 commented 3 years ago

Hi there, was there any movement on the flagger support? We are currently using Consul for pod to pod encryption. But we are also moving towards a gitops solutions and therefore want to use flagger, prometheus, fluxcd, etc. to orchestrate deployments. Without such a support, we will have to start evaluation other ServiceMesh solutions.

nicholasjackson commented 3 years ago

Hey, @ssimard24, @oleksiyp I have been chatting with Stephan from Weave and have created a new SMI controller that allows you to use Flagger without modifications. This is still pretty early stages but it works great for Flaggers requirement round Service Splitting. Personally, I love flagger and would not deploy without it, we are excited to bring this to Consul.

https://github.com/nicholasjackson/consul-smi-controller

I have also started to write up this information, and plan to produce a video within the next week to show how this all works.

https://github.com/nicholasjackson/consul-canary-deployment

Kind regards,

Nic

ssimard24 commented 3 years ago

Great news Nicholas. Thanks for your help.

ssimard24 commented 2 years ago

Hey Nicholas was there any more movement on the topic of Flagger? We are ready to start our proof of concept.

shubhamsre commented 1 year ago

@nicholasjackson could you please let know if there has been a progress on the feature, and is it still in POC phase or in GA. Please point to the official docs, if any. Thanks.

hashicorp / consul