Open arinhouck opened 1 year ago
From what I can gather, zookeeper is the only option with Amazon MSK for managing ACLs. So it doesn't seem this is supporting Amazon MSK at all if I am not mistaken. Has anyone got this working with MSK?
@arinhouck, I was able to create ACL over MKS using plaintext
According to your code I can suggest turning on skip_tls_verify
attribute to true, and trying again.
I'd suggest setting up the provider in the following way.
provider "kafka" {
bootstrap_servers = split(",", var.servers)
tls_enabled = false
skip_tls_verify = true
}
@hugolesta if create ACL over MSK using SSL, It's failed. so do you have solution for that? As we know, considering security requirement, in common, using SSL is required in Production Environment.thanks
@arinhouck have any update? I also encounter the same issue with you.
@arinhouck, I was able to create ACL over MKS using plaintext
According to your code I can suggest turning on
skip_tls_verify
attribute to true, and trying again.I'd suggest setting up the provider in the following way.
provider "kafka" { bootstrap_servers = split(",", var.servers) tls_enabled = false skip_tls_verify = true }
Plaintext on which server urls? I'd assume bootstrap ones. You sure you are using zookeeper? As from what I understand the library maps to --bootstrap-server ...
it doesn't use --zookeeper.connect=...
. Is your cluster public as well?
@qq304635576 I ditched SCRAM and zookeeper. I ended up using IAM Auth which allows you to bypass zookeeper. I used https://github.com/devshawn/kafka-gitops using the following script from this comment to setup IAM auth.
https://github.com/devshawn/kafka-gitops/issues/82#issuecomment-985287979
@arinhouck Actually, Looks like no need to care about zookeeper, I can modify ACL by a client app named "offset explorer 2" without configuring Zookeeper over MSK using SASL_SSL manually. IAM Auth is a new feature, which is owned by AWS MSK only. considering MSK as bus info channel, It should be the most widely compatible with apps for auth. that's why I chose SASL_SSL. moreover, will check your recommendation, maybe I will change to IAM auth in future. thanks.
@arinhouck Good news! I tested again via SASL_SSL & Port:9096. It woks. (1) Set up as below:
provider "msk" { bootstrap_servers = var.msk_kafka_brokers tls_enabled = true skip_tls_verify = true sasl_username = local.raw_data.username sasl_password = local.raw_data.password sasl_mechanism = "scram-sha512" }
(2)Terraform output:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
Terraform will perform the following actions:
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.
Enter a value: yes
kafka_acl.brokertopic: Creating... kafkaacl.brokertopic: Creation complete after 2s [id=User:broker|*|All|Allow|Topic|TEST|Prefixed] Releasing state lock. This may take a few moments...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
I am able to run the following within my Codebuild instance within Amazon VPC using Kafka CLI to change Zookeeper ACL and topic.
Create ACL
Create Topic
However, when configuring via terraform it always returns this error:
using
where var.servers = "some-domain.us-east-1.amazonaws.com:2181" and var.scram_user = "scramuser".
So I have special flags assigned for zookeeper config as you can see in the commands above. Does this library only allow bootstrap servers? I don't see any documentation for using zookeeper config.
Following practices from following AWS docs: https://docs.aws.amazon.com/msk/latest/developerguide/msk-acls.html https://docs.aws.amazon.com/msk/latest/developerguide/mkc-create-topic.html
My goal is to streamline this process using terraform instead of managing sh scripts. However unsure if that is achievable. I actually still don't understand what the difference is for bootstrap server vs zookeeper server. I guess zookeeper is like a middleware. I doesn't seem Amazon provides any other option other than zookeeper for modifying topics and ACL in documentation.
EDIT: I see zookeeper isn't supported according to discussions in past issues of this repo.
I have tried SASL SCRAM-512 which works on public endpoint for consumer and producer on making connection. However for the private endpoint when running terraform apply through this library with SCRAM configured it doesn't seem to work even with all traffic allowed through VPC and in a VPC enabled environment (same instance that private zookeeper dns worked for).
I have verified in the variables using codebuild-breakpoint that the variables are correctly assigned for username and password.
provider "kafka" { bootstrap_servers = split(",", var.servers) tls_enabled = false sasl_username = var.scram_username sasl_password = var.scram_password sasl_mechanism = "scram-sha512" }