search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.22k stars 493 forks source link

AWS EKS: unable to connect to the cluster with create_vpc=false and existing VPC #514

Closed george24601 closed 5 years ago

george24601 commented 5 years ago

Bug Report

What version of Kubernetes are you using? AWS EKS 1.12 on us-east-2

What did you do?

  1. Create a new tidb cluster on EKS following the README with create_vpc tf variable set to true
  2. Verified that creation is successful by logging into tidb and executing queries
  3. Create a second tidb cluster with a copy of the terraform in a separate directory (so that tfstate files don't interfere with each other), this time with create_vpc = false, i.e., my tfvar files looks like
pd_instance_type = "c5d.large"
tikv_instance_type = "c5d.large"
tidb_instance_type = "c4.large"
monitor_instance_type = "c5.large"

pd_count = 1
tikv_count = 1
tidb_count = 1

cluster_name = "rep_george"
tikv_root_volume_size = "50"

region = "us-east-2"
create_vpc=false
vpc_id="$VPC_ID_CREATED_IN_STEP_ONE"
subnets=$SUBNET_IDS_CREATED_IN_STEP_ONE

What did you expect to see? the second tidb cluster will be created inside the same VPC, but on a second EKS cluster

What did you see instead? Unable to connect to the bastion host, because it went to the private subnet

From the tf code, seems that when create_vpc=false, we should use separate subnet id variables for bastion and EKS, so that bastion goes to the public subnet, EKS private? https://github.com/pingcap/tidb-operator/blob/master/deploy/aws/main.tf#L72 https://github.com/pingcap/tidb-operator/blob/master/deploy/aws/main.tf#L89

AstroProfundis commented 5 years ago

Hello, thank you for the report, we are looking into this issue and will update later. Your patience will be appreciated.

AstroProfundis commented 5 years ago

Reusing existing VPC and subnets that created by Terraform for another EKS cluster is not supported now due to various limitations. However, it is possible to make the second cluster deployed by manually adding tag kubernetes.io/cluster/<second_cluster_name>=shared to the subnets of the first cluster (if that tag doesn't already exist).

Note that this is not recommend and may cause dependency issues when trying to modify or destroy these resources.

We have submitted PR to correctly put resources into public and private subnets, and to make the docs more clear on this kind of situation.