Configure mTLS on Schema Registry

[vepo]

I'd like to see documentation for Schema Registry.

This software can:

• [X] Accept TLS connections
• [X] Authenticate clients with TLS
• [X] Request TLS connections with client certificates

[mmalone]

I just skimmed the docs. Looks like this is built on Kafka, but it exposes its own APIs for schema management. It looks like it also (optionally?) talks to Zookeeper? So there are at least three connections that could theoretically be secured using [m]TLS (client<->schema registry, schema registry<->kafka, and kafka<->zookeeper).

The existing kafka docs may help get you started. Skimming through the security configuration docs for Schema Registry, it looks like the configuration is very similar to Kafka. So my guess is the Kafka instructions translate pretty well to this use case.

This seems like it will be challenging for us to document ourselves since we aren't familiar with the software and there are a lot of moving parts we'd have to spin up just to get a test infrastructure in place. Hopefully, the kafka docs help for now. If you have any specific questions / run into any problems I'd be happy to help as best I can.

[vepo]

Sure. I understand. I'm trying to configure it, but it has been a difficult task. I create the ticket as an suggestion.

If this can help, I can share the solution when I find one, but, for now, I have create an issue on Schema Registry repo.

[mmalone]

Sorry to hear that. I do appreciate the suggestion, just wanted to be upfront about how much we can help and how quickly.

I just took a look at the issue you opened with confluent. It definitely looks like there's something wrong with the client certificate configuration: Schema Registry seems to be either 1) not presenting a certificate, or 2) presenting an expired certificate to Kafka. I found this stack overflow post that indicates the error you're seeing may be due to an expired client certificate. Looks likekeytool -list -keystore client.keystore.jks -v may provide the information we need to debug further. If you've been fighting with this for a while, this could very well be your issue since (by default) step-ca issues short-lived certificates that expire after 24 hours.

There's a lot of TLS configuration going on, which isn't surprising given all of the components, but it is definitely possible that there's something misconfigured there. The confluent folks would be better equipped to answer that. Assuming the truststore/keystore configuration is all correct, there there may be some other issue with the contents of your key stores, or with the certificates themselves. If you could share the process you're using to obtain certificates and construct your keystores I may be able to help debug further.

[vepo]

Sure!

I'm using the script below. I have little experience with encryption, so any help is welcome.

export MSYS_NO_PATHCONV=1
cd certs
rm ca-* cert-* kafka*
export PASSWORD=password
echo $PASSWORD > password

export HOSTNAME=kafka
openssl req -new -newkey rsa:4096 -days 365 -x509 -subj "/CN=Demo-Kafka" -keyout ca-key -out ca-cert -nodes
keytool -genkey -keyalg RSA -keystore kafka.server.keystore.jks -validity 365 -storepass $PASSWORD -keypass $PASSWORD -dname "CN=$HOSTNAME" -storetype pkcs12
keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass $PASSWORD -keypass $PASSWORD
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-file-signed -days 365 -CAcreateserial -passin pass:$PASSWORD
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore kafka.server.keystore.jks -import -file cert-file-signed -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt

rm ca-* cert-* schema-registry*
export HOSTNAME=schema-registry
openssl req -new -newkey rsa:4096 -days 365 -x509 -subj "/CN=Demo-Schema-Registry" -keyout ca-key -out ca-cert -nodes
keytool -genkey -keyalg RSA -keystore schema-registry.server.keystore.jks -validity 365 -storepass $PASSWORD -keypass $PASSWORD -dname "CN=$HOSTNAME" -storetype pkcs12
keytool -keystore schema-registry.server.keystore.jks -certreq -file cert-file -storepass $PASSWORD -keypass $PASSWORD
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-file-signed -days 365 -CAcreateserial -passin pass:$PASSWORD
keytool -keystore schema-registry.server.keystore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore schema-registry.server.keystore.jks -import -file cert-file-signed -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore schema-registry.server.truststore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore schema-registry.client.truststore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt

rm ca-* cert-* zookeeper*
export HOSTNAME=zookeeper
openssl req -new -newkey rsa:4096 -days 365 -x509 -subj "/CN=Demo-Zookeeper" -keyout ca-key -out ca-cert -nodes
keytool -genkey -keyalg RSA -keystore zookeeper.server.keystore.jks -validity 365 -storepass $PASSWORD -keypass $PASSWORD -dname "CN=$HOSTNAME" -storetype pkcs12
keytool -keystore zookeeper.server.keystore.jks -certreq -file cert-file -storepass $PASSWORD -keypass $PASSWORD
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-file-signed -days 365 -CAcreateserial -passin pass:$PASSWORD
keytool -keystore zookeeper.server.keystore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore zookeeper.server.keystore.jks -import -file cert-file-signed -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore zookeeper.server.truststore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt
keytool -keystore zookeeper.client.truststore.jks -alias CARoot -import -file ca-cert -storepass $PASSWORD -keypass $PASSWORD -noprompt

#ref https://medium.com/jinternals/kafka-ssl-setup-with-self-signed-certificate-part-1-c2679a57e16c

[vepo]

This keystore is not for production, I'm working on a project the uses Kafka/Schema Registry outside our control.

[mmalone]

It looks like you may be issuing Schema Registry's certificate from the wrong CA. Honestly, for this use case you can probably use the same root CA certificate for both Kafka and Schema Registry. Instead of deleting your ca-cert and ca-key in between steps, try using the same CA throughout (i.e., remove the rm ca-* in between each step and the first openssl command for Schema Registry & Zookeeper). That should at least make things a little easier to configure for the moment.

You may also need to configure a "client keystore" for Schema Registry. This is where I'm particularly out of my depths, since I don't know much about Kafka configuration or about Java / PKCS#12 keystore/truststores. But you're gonna need to configure Schema Registry to use a client certificate that was issued by the CA that Kafka trusts. I don't see where that's happening. It's possible that Schema Registry will use the same truststore/keystore for client and server authentication, but it looks like the one certificate you're giving to Schema Registry is being issued off of a CA that Kafka isn't configured to trust.

[mingmingshiliyu]

It looks like you may be issuing Schema Registry's certificate from the wrong CA. Honestly, for this use case you can probably use the same root CA certificate for both Kafka and Schema Registry. Instead of deleting your ca-cert and ca-key in between steps, try using the same CA throughout (i.e., remove the rm ca-* in between each step and the first openssl command for Schema Registry & Zookeeper). That should at least make things a little easier to configure for the moment.

You may also need to configure a "client keystore" for Schema Registry. This is where I'm particularly out of my depths, since I don't know much about Kafka configuration or about Java / PKCS#12 keystore/truststores. But you're gonna need to configure Schema Registry to use a client certificate that was issued by the CA that Kafka trusts. I don't see where that's happening. It's possible that Schema Registry will use the same truststore/keystore for client and server authentication, but it looks like the one certificate you're giving to Schema Registry is being issued off of a CA that Kafka isn't configured to trust.

hey,what about mils between schema registry and client?do you have any best practice?or only configure mtls in nginx to proxy schema registry?