search

Mongey / terraform-provider-kafka

Terraform provider for managing Apache Kafka Topics + ACLs
MIT License
517 stars 129 forks source link

connect to AWS MSK cluster #158

Open danschreifels2019 opened 3 years ago

danschreifels2019 commented 3 years ago

I'm getting the following error when trying to connect to the MSK bootstrap servers. "Error: rpc error: code = Unavailable desc = transport is closing"

since I'm trying to connect outside of AWS, I'm using the grepplabs/kafka-proxy to be able to connect outside of AWS. this work's fine from my workstation. however when running the kafka-provider in terraform I get the error message above.

I'm using TLS on the MSK cluster and in the terraform code I set tls_enabled = true and skip_tls_verify = false since I can't get the certs from AWS for MSK cluster.

any help on this would be great. Is there a way to pass the command.config in an ENV variable?

this seems to be a configuration issue in the connection or transport setting with the provider.

TanmayaAPYL commented 3 years ago

Same here. I get 

2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: panic: runtime error: invalid memory address or nil pointer dereference
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xeeed6c]
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: goroutine 42 [running]:
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: github.com/Mongey/terraform-provider-kafka/kafka.NewClient(0xc000304090, 0xc0002b5360, 0xc00026e198, 0x42dd4a)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/work/terraform-provider-kafka/terraform-provider-kafka/kafka/client.go:31 +0x1bc
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: github.com/Mongey/terraform-provider-kafka/kafka.(*LazyClient).init.func1()
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/work/terraform-provider-kafka/terraform-provider-kafka/kafka/lazy_client.go:21 +0x40
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: sync.(*Once).doSlow(0xc000466450, 0xc0002a3640)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /opt/hostedtoolcache/go/1.13.15/x64/src/sync/once.go:66 +0xe3
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: sync.(*Once).Do(...)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /opt/hostedtoolcache/go/1.13.15/x64/src/sync/once.go:57
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: github.com/Mongey/terraform-provider-kafka/kafka.(*LazyClient).init(0xc000466450, 0xc0001564d0, 0x70)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/work/terraform-provider-kafka/terraform-provider-kafka/kafka/lazy_client.go:20 +0x2a3
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: github.com/Mongey/terraform-provider-kafka/kafka.(*LazyClient).ReadTopic(0xc000466450, 0xc0002685a0, 0x18, 0x0, 0x0, 0xc0002a3f70, 0x1cbc280, 0xc0002a3860, 0xc0002a3860)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/work/terraform-provider-kafka/terraform-provider-kafka/kafka/lazy_client.go:41 +0x2f
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: github.com/Mongey/terraform-provider-kafka/kafka.topicRead(0xc0001564d0, 0x11098a0, 0xc000466450, 0xc0001564d0, 0x0)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/work/terraform-provider-kafka/terraform-provider-kafka/kafka/resource_kafka_topic.go:181 +0xcc
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000128a00, 0xc0002c8190, 0x11098a0, 0xc000466450, 0xc00026e208, 0x0, 0x0)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.0.0/helper/schema/resource.go:455 +0x119
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc0000cea20, 0x14bb420, 0xc0002b36b0, 0xc0000d5e50, 0xc0000cea20, 0xc0002b36b0, 0xc0001cda80)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.0.0/internal/helper/plugin/grpc_provider.go:525 +0x3d8
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadResource_Handler(0x1171ce0, 0xc0000cea20, 0x14bb420, 0xc0002b36b0, 0xc000306de0, 0x0, 0x14bb420, 0xc0002b36b0, 0xc0002b80c0, 0xb1)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.0.0/internal/tfplugin5/tfplugin5.pb.go:3153 +0x217
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000112000, 0x14ca060, 0xc000447500, 0xc000266600, 0xc00012a810, 0x1c8fa50, 0x0, 0x0, 0x0)
2020-12-03T16:16:59.985Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:995 +0x460
2020-12-03T16:16:59.986Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: google.golang.org/grpc.(*Server).handleStream(0xc000112000, 0x14ca060, 0xc000447500, 0xc000266600, 0x0)
2020-12-03T16:16:59.986Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:1275 +0xd97
2020-12-03T16:16:59.986Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000483ee0, 0xc000112000, 0x14ca060, 0xc000447500, 0xc000266600)
2020-12-03T16:16:59.986Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:710 +0xbb
2020-12-03T16:16:59.986Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-12-03T16:16:59.986Z [DEBUG] plugin.terraform-provider-kafka_v0.2.10:   /home/runner/go/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:708 +0xa1
2020/12/03 16:16:59 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/12/03 16:16:59 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing

OS - Ubuntu 18.04.5 LTS

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/archive v2.0.0
+ provider registry.terraform.io/hashicorp/aws v3.19.0
+ provider registry.terraform.io/hashicorp/helm v1.3.2
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.3
+ provider registry.terraform.io/hashicorp/local v1.4.0
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/hashicorp/random v2.3.1
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/mongey/kafka v0.2.10
mikalai-t commented 3 years ago

Same here. But I managed to fix it. In my case there was something like:

data "aws_ssm_parameter" "bootstrap_brokers_tls" {
  count = length(var.topics) > 0) ? 1 : 0

  depends_on = [
    var.module_depends_on
  ]

  name = "/<some path>/bootstrap-brokers-tls"
}

provider "kafka" {
  bootstrap_servers = split(",", join("", data.aws_ssm_parameter.bootstrap_brokers_tls[*].value))

  tls_enabled     = true
  skip_tls_verify = false
}

depends_on block in the data source caused a "segmentation fault" exception with a stack-trace and so on. When I removed it the module started working.

I'd recommend:

  1. Enable debug with export TF_LOG=DEBUG
  2. Run terraform plan or apply again
  3. Check output for the [INFO] configuring bootstrap_servers message.
  4. If its value is something like: {<nil> 120 **** **** yada yada... } - it might be similar case what means your provider's config is incorrect (instead of \<nil> there must be a hexadecimal sequence)
andormarkus commented 3 years ago

Hi All,

We choose different approach to configure our MSK cluster: OpenVPN server. We created an OpenVPN server on a public subnet which reach our private MSK cluster. The OpenVPN server only used for CI/CD purposes. On the CI/CD pipeline we are opening a VPN connection to our AWS VPC and deployment works without any issue.

NOTE: not all CI/CD provider supports docker privileged containers or VM executors

provider "kafka" {
  bootstrap_servers = split(",", aws_msk_cluster.kafka_main.bootstrap_brokers)

  tls_enabled     = false
  skip_tls_verify = true
}