scholzj
commented
3 years ago Can you describe the steps for reproducing this? It does not seem to be happening to me.
Closed wiegandf closed 3 years ago
scholzj
commented
3 years ago Can you describe the steps for reproducing this? It does not seem to be happening to me.
scholzj
commented
3 years ago I tried it with the following YAML:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-source
spec:
kafka:
version: 2.7.0
replicas: 3
listeners:
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
# authorization:
# type: simple
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.7"
inter.broker.protocol.version: "2.7"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: true
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: true
entityOperator:
topicOperator: {}
userOperator: {}
---
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-target
spec:
kafka:
version: 2.7.0
replicas: 3
listeners:
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
# authorization:
# type: simple
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.7"
inter.broker.protocol.version: "2.7"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: true
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: true
entityOperator:
topicOperator: {}
userOperator: {}
---
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaUser
metadata:
name: my-source-mirror-maker
labels:
strimzi.io/cluster: my-source
spec:
authentication:
type: tls
---
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaUser
metadata:
name: my-target-mirror-maker
labels:
strimzi.io/cluster: my-target
spec:
authentication:
type: tls
---
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
name: my-mirror-maker-2
spec:
version: 2.7.0
replicas: 1
connectCluster: "my-target"
clusters:
- alias: "my-source"
bootstrapServers: my-source-kafka-bootstrap:9093
tls:
trustedCertificates:
- secretName: my-source-cluster-ca-cert
certificate: ca.crt
authentication:
certificateAndKey:
certificate: user.crt
key: user.key
secretName: my-source-mirror-maker
type: tls
- alias: "my-target"
bootstrapServers: my-target-kafka-bootstrap:9093
tls:
trustedCertificates:
- secretName: my-target-cluster-ca-cert
certificate: ca.crt
authentication:
certificateAndKey:
certificate: user.crt
key: user.key
secretName: my-target-mirror-maker
type: tls
config:
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
mirrors:
- sourceCluster: "my-source"
targetCluster: "my-target"
sourceConnector:
config:
replication.factor: 1
offset-syncs.topic.replication.factor: 1
sync.topic.acls.enabled: "false"
heartbeatConnector:
config:
heartbeats.topic.replication.factor: 1
checkpointConnector:
config:
checkpoints.topic.replication.factor: 1
topicsPattern: ".*"
groupsPattern: ".*"But all seems to work fine for me. It would be great if you could provide a full log or some more detailed steps to reproduce. Especially the log from the container start when it generates the keystores would be useful.
wiegandf
commented
3 years ago Hi, thanks so much for your support.
I can confirm that it seems to work when I spin up a new cluster with the operator and use self-generated certificates.
The certs which are not working are generated by terraform and are issued by our private AWS CA (aws pca). https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acm_certificate is used to generate the certificates . https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acmpca_certificate_authority for the CA (with RSA_2048 and SHA256WITHRSA config).
The cert is saved to our vault with some python script:
certificate["PrivateKey"] = private_key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.BestAvailableEncryption(passphrase_encoded),
).decode("ascii")Before creating the k8s secret for MM2, I'll decrypt the key manually via
openssl rsa -in in.key -out out.key
I've attached the full logs to the post and meanwhile look how to better reproduce this. log.txt
scholzj
commented
3 years ago It looks like for some reason, the connector config file with the passwords did not generate. This is in your log:
Creating connector configuration:
While this is in mine:
Creating connector configuration:
# TLS / SSL
ssl.truststore.password=[hidden]
ssl.keystore.password=[hidden]It looks like there is a bug in how the Mirror Maker 2 deployments handles the situation when you want to use the default TLS certificates from the JVM instead of passing your own. So basically this part is causing the issue.
tls:
trustedCertificates: []I will have a look at it and see how it can be fixed. But in the meantime, as a workaround, you would need to create a secret with the public key of the CA used by your cluster and use it in the tls section:
tls:
trustedCertificates:
- secretName: my-ca-secret
certificate: ca.crtI opened a PR for this. I'm 99% sure it fixes the issue. But I do not have the exactly same environment with server certs signed by public CA and TLS client authentication. So if you want and have some test cluster to try it in, I can provide you with the images to try it.
Describe the bug I am trying to use MirrorMaker2 with strimzi operator to mirror to an AWS MSK cluster. I am using TLS authentication on MSK side and the certificates are referenced in the YAML below. It looks like MM2 is creating the keystore correctly but then fails to read from it.
When I try to open the keystore, it works when I don't enter any password:
When I enter any arbitrary password I'll get an error similar to MM2 stacktrace:
Environment (please complete the following information):
YAML files and logs
Additional context As it seems that
kafka_mirror_maker_tls_prepare_certificates.shis creating the keystores, I also tried to set CERTS_STORE_PASSWORD but it gave me the same error.