search

gardener-attic / kubify

Terraform Template to Setup a Kubernetes Cluster on OpenStack/AWS/Azure
Other
147 stars 31 forks source link

Failed to deploy kubify on OpenStack #64

Closed jiangytcn closed 5 years ago

jiangytcn commented 5 years ago

Hi,

I create a customized dns module to use openstack designate ans dns provider, but after create openstack related variables, when running terraform plan variant the deployment still reaching out to aws ec2. I only use aws s3 for etcd backup

Here's the changes https://github.com/jiangytcn/kubify/commit/630a2b01a5b9a70c46382545b604e21934dc1b29


$ cat terraform.tfvars | grep -v '#' | grep -v '^$'                                                                    
os_user_name = "admin"
os_password = "ae226d1f8b27c60b31088"
os_auth_url = "http://172.29.236.100:5000/v3"
os_tenant_name = "demo"
os_domain_name = "default"
os_region = "RegionOne"
os_fip_pool_name = "public"
os_lbaas_provider = "haproxy"
os_az = "nova"
event_ttl = "168h0m0s"
os_vpc_cidr = "10.251.0.0/16"
cluster_name = "management"
cluster_type = "eval"
versions = {
  image_name = "coreos-1688.5.3"
}
dns = {
  domain_name = "lab.yacloud.int"
  dns_type = "designate"
  hosted_zone_id = "5ad92a47-def0-45af-8e6d-ed35f6a1fee0"
  access_key = "dummy"
  secret_key = "dummy"
}
master = {
  count = 3
  volume_size = 50
}
worker = {
  count = 3
  volume_size = 50
}
etcd_backup = {
  "access_key" = "XXXX"
  "region" = "ap-northeast-1"
  "secret_key" = "XXXX"
  "storage_type" = "s3"
}
addons = {
  "dashboard" = {
    "app_name" = "kubernetes-dashboard"
  }
  "nginx-ingress" = {
  }
}
dashboard_creds = "admin"
deploy_tiller = false
oidc_issuer_subdomain = "identity.ingress"
oidc_client_id = "kube-kubectl"
oidc_username_claim = "email"
oidc_groups_claim = "groups"
subnet_cidr = "10.251.128.0/17"
service_cidr = "10.241.0.0/17"
pod_cidr = "10.241.128.0/17"
selfhosted_etcd = "false"

Failures in terraform 

module.instance.null_resource.master_setup - *terraform.NodePlannableResourceInstance
2019/01/17 05:12:52 [TRACE] Graph after step *terraform.RootTransformer:

module.instance.null_resource.master_setup - *terraform.NodePlannableResourceInstance
2019/01/17 05:12:52 [DEBUG] Resource state not found for "module.instance.local_file.reset_bootkube": module.instance.local_file.reset_bootkube
2019/01/17 05:12:52 [DEBUG] ReferenceTransformer: "module.instance.local_file.reset_bootkube" references: []
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:52 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: ---[ RESPONSE ]--------------------------------------
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: HTTP/1.1 403 Forbidden
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Connection: close
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Content-Length: 306
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Content-Type: text/xml
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Date: Thu, 17 Jan 2019 05:12:51 GMT
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: X-Amzn-Requestid: 8a1734aa-1a16-11e9-9b9a-eba3b07524ac
2019-01-17T05:12:52.112Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: -----------------------------------------------------
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:52 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:   <Error>
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <Type>Sender</Type>
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <Code>InvalidClientTokenId</Code>
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <Message>The security token included in the request is invalid.</Message>
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:   </Error>
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:   <RequestId>8a1734aa-1a16-11e9-9b9a-eba3b07524ac</RequestId>
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: </ErrorResponse>
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:52 [DEBUG] [aws-sdk-go] DEBUG: Validate Response sts/GetCallerIdentity failed, not retrying, error InvalidClientTokenId: The security token included in the request is invalid.
2019-01-17T05:12:52.113Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  status code: 403, request id: 8a1734aa-1a16-11e9-9b9a-eba3b07524ac
2019/01/17 05:12:52 [ERROR] root: eval: *terraform.EvalConfigProvider, err: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 8a1734aa-1a16-11e9-9b9a-eba3b07524ac
2019/01/17 05:12:52 [ERROR] root: eval: *terraform.EvalSequence, err: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 8a1734aa-1a16-11e9-9b9a-eba3b07524ac
2019/01/17 05:12:52 [ERROR] root: eval: *terraform.EvalOpFilter, err: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 8a1734aa-1a16-11e9-9b9a-eba3b07524ac
2019/01/17 05:12:52 [ERROR] root: eval: *terraform.EvalSequence, err: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 8a1734aa-1a16-11e9-9b9a-eba3b07524ac
2019/01/17 05:12:52 [TRACE] [walkPlan] Exiting eval tree: provider.aws.route53
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:52 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: ---[ RESPONSE ]--------------------------------------
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: HTTP/1.1 200 OK
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Connection: close
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Content-Length: 406
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Content-Type: text/xml
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Date: Thu, 17 Jan 2019 05:12:52 GMT
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: X-Amzn-Requestid: 8a5e768b-1a16-11e9-9b72-73f4bff84a99
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: -----------------------------------------------------
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:52 [DEBUG] [aws-sdk-go] <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:   <GetCallerIdentityResult>
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <Arn>arn:aws:iam::xxxxxxx:user/jiangytcn</Arn>
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <UserId>xxxx</UserId>
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <Account>xxxx</Account>
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:   </GetCallerIdentityResult>
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:   <ResponseMetadata>
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <RequestId>8a5e768b-1a16-11e9-9b72-73f4bff84a99</RequestId>
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:   </ResponseMetadata>
2019-01-17T05:12:52.582Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: </GetCallerIdentityResponse>
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:52 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeAccountAttributes Details:
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: POST / HTTP/1.1
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Host: ec2.ap-northeast-1.amazonaws.com
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: User-Agent: aws-sdk-go/1.16.16 (go1.11.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.9-beta1
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Content-Length: 87
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=xxxxxxx/20190117/ap-northeast-1/ec2/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=e6cfb85b85654a8ccfced1d8d64963e38cd4aafd0e0298fb37c0dcabea1deb43
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: X-Amz-Date: 20190117T051252Z
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Accept-Encoding: gzip
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Action=DescribeAccountAttributes&AttributeName.1=supported-platforms&Version=2016-11-15
2019-01-17T05:12:52.583Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: -----------------------------------------------------
2019-01-17T05:12:53.115Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:53 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeAccountAttributes Details:
2019-01-17T05:12:53.115Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: ---[ RESPONSE ]--------------------------------------
2019-01-17T05:12:53.115Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: HTTP/1.1 200 OK
2019-01-17T05:12:53.115Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Connection: close
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Content-Length: 540
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Content-Type: text/xml;charset=UTF-8
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Date: Thu, 17 Jan 2019 05:12:52 GMT
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: Server: AmazonEC2
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: -----------------------------------------------------
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:53 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: <DescribeAccountAttributesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <requestId>d408d2ed-58a9-43cf-a9e4-3a7a27d4205f</requestId>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     <accountAttributeSet>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:         <item>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:             <attributeName>supported-platforms</attributeName>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:             <attributeValueSet>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:                 <item>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:                     <attributeValue>VPC</attributeValue>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:                 </item>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:             </attributeValueSet>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:         </item>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:     </accountAttributeSet>
2019-01-17T05:12:53.116Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: </DescribeAccountAttributesResponse>
2019/01/17 05:12:53 [DEBUG] Resource state not found for "module.instance.module.seed.aws_s3_bucket.s3_etcd_backup": module.instance.module.seed.aws_s3_bucket.s3_etcd_backup
2019/01/17 05:12:53 [TRACE] Graph after step *terraform.AttachStateTransformer:

module.instance.module.seed.aws_s3_bucket.s3_etcd_backup - *terraform.NodePlannableResourceInstance
2019/01/17 05:12:53 [DEBUG] ReferenceTransformer: "module.instance.module.seed.aws_s3_bucket.s3_etcd_backup" references: []
2019/01/17 05:12:53 [DEBUG] plugin: waiting for all plugin processes to complete...

Error: Error running plan: 1 error(s) occurred:

* provider.aws.route53: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 8a1734aa-1a16-11e9-9b9a-eba3b07524ac

2019-01-17T05:12:53.134Z [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 2019/01/17 05:12:53 [ERR] plugin: plugin server: accept unix /tmp/plugin363027272: use of closed network connection
2019-01-17T05:12:53.134Z [DEBUG] plugin.terraform-provider-tls_v1.2.0_x4: 2019/01/17 05:12:53 [ERR] plugin: plugin server: accept unix /tmp/plugin088484845: use of closed network connection
2019-01-17T05:12:53.134Z [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2019/01/17 05:12:53 [DEBUG] plugin: waiting for all plugin processes to complete...
2019-01-17T05:12:53.134Z [DEBUG] plugin.terraform-provider-openstack_v1.13.0_x4: 2019/01/17 05:12:53 [ERR] plugin: plugin server: accept unix /tmp/plugin188386594: use of closed network connection
2019-01-17T05:12:53.134Z [DEBUG] plugin: plugin process exited: path=/landscape/.terraform/plugins/linux_amd64/terraform-provider-local_v1.1.0_x4
2019-01-17T05:12:53.134Z [DEBUG] plugin: plugin process exited: path=/landscape/.terraform/plugins/linux_amd64/terraform-provider-archive_v1.1.0_x4
2019-01-17T05:12:53.134Z [DEBUG] plugin.terraform-provider-template_v2.0.0_x4: 2019/01/17 05:12:53 [ERR] plugin: plugin server: accept unix /tmp/plugin938372641: use of closed network connection
2019-01-17T05:12:53.135Z [DEBUG] plugin: plugin process exited: path=/landscape/.terraform/plugins/linux_amd64/terraform-provider-openstack_v1.13.0_x4
2019-01-17T05:12:53.134Z [DEBUG] plugin.terraform: file-provisioner (internal) 2019/01/17 05:12:53 [DEBUG] plugin: waiting for all plugin processes to complete...
2019-01-17T05:12:53.135Z [DEBUG] plugin: plugin process exited: path=/landscape/.terraform/plugins/linux_amd64/terraform-provider-aws_v1.55.0_x4
2019-01-17T05:12:53.134Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform
2019-01-17T05:12:53.134Z [DEBUG] plugin.terraform-provider-random_v2.0.0_x4: 2019/01/17 05:12:53 [ERR] plugin: plugin server: accept unix /tmp/plugin446398090: use of closed network connection
2019-01-17T05:12:53.134Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform
2019-01-17T05:12:53.135Z [DEBUG] plugin: plugin process exited: path=/landscape/.terraform/plugins/linux_amd64/terraform-provider-random_v2.0.0_x4
2019-01-17T05:12:53.134Z [DEBUG] plugin: plugin process exited: path=/landscape/.terraform/plugins/linux_amd64/terraform-provider-template_v2.0.0_x4
2019-01-17T05:12:53.135Z [DEBUG] plugin: plugin process exited: path=/landscape/.terraform/plugins/linux_amd64/terraform-provider-null_v1.0.0_x4
2019-01-17T05:12:53.136Z [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2019/01/17 05:12:53 [DEBUG] plugin: waiting for all plugin processes to complete...
2019-01-17T05:12:53.136Z [DEBUG] plugin: plugin process exited: path=/landscape/.terraform/plugins/linux_amd64/terraform-provider-tls_v1.2.0_x4
2019-01-17T05:12:53.139Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform

Btw, is it possible to disable etcd backup to s3 ?

jiangytcn commented 5 years ago

close it. the issue is still related to dns registration under openstack environment