Simplified productized HA offering

[AnkurMundhra]

What we want:

• Multi-node installation of A2 (front-end and back-end) - done
• Failover support (no manual intervention required to shift form hot to cold cluster)
• Ability to connect to multiple databases (PostgreSQL cluster, ES cluster) - doc required
• Blue-green deployment: Sequential update when a patch is given (zero downtime)
• Externally configurable (and user provided) load balancer (front-end), databases (ES, RDS, S3) - tested with HAProxy, Nginx - Doc required
• Enhance smoke tests - provide description, what are improvement required?
• Self-serve usage - deployment should be easy
• Terraform support for on-prem and cloud-providers (start with AWS, then GCP, Azure) - On-prem, AWS are possible now.
• Across multiple availability zone/ data centres. - AWS supported currently
• Auto-scaling - on nodes, managed services for ES
• Centralized config management across clusters (front end dex configs, example bind_dn, bind_password)
• For 200k nodes (for load test).
• To benchmark performance, we need to update document after load testing.
• [Audit data should not record of system down]
• Need to come up with a flow for smoother deployment.
• Backup & Restoration. - doc required
• Simple monitoring and alerting system: metrics from all services to flow to Prometheus
• Option to upgrade front-end automate ( and front-end Chef Infra server )
• Nginx-based rate limiting

Out of scope:

• Azure platform support for A2 HA

Existing issues: We need to close them.

https://github.com/chef/customer-bugs/issues/231	Chef Backend Cluster stability issues #231

Aha! Link: https://chef.aha.io/epics/AUTO-E-15

[AnkurMundhra]

Comment added by Ankur Mundhra in Aha! - View

Can refer to https://chefio.slack.com/archives/C01NG0QRHEH/p1614058311003600.

[punitmundra]

1. Deployment over the Azure with exiting_node architecture .
2. Need to try Backup and restore the on storage bucket(Azure and GCP).