Apache Kafka, SSL and TLS configuration in grpc spring boot

[DevShivmohan]

Create openssl certificate for localhost

//Generate CA's private key openssl genrsa -des3 -out ca.key.pem 2048

//create CA's self-signed certificate openssl req -x509 -new -nodes -key ca.key.pem -sha256 -days 365 -out localhost.cert.pem

//create private key for server openssl genrsa -out localhost.key 2048

//create certificate signing request (CSR) openssl req -new -key localhost.key -out localhost.csr

//Use CA's private key to sign web server's CSR and get back the signed certificate openssl x509 -req -in localhost.csr -CA localhost.cert.pem -CAkey ca.key.pem -CAcreateserial -out localhost.crt -days 365

//convert server private key in PKCS8 standard(gRPc expects) openssl pkcs8 -topk8 -nocrypt -in localhost.key -out localhost.pem

Demo with live screenshots

[DevShivmohan]

Spring boot grpc configuration with TLS security

• Configuration in grpc-client as a microservice, consider cert directory and application.yml
• 
• Configuration with grpc-service as a microservice, consider cert directory and application.yml
• 
• Note replace the file paths as per your configurations.

[DevShivmohan]

Install Apache Kafka and configure it in Ubuntu

• Goto the official web site https://kafka.apache.org/downloads.
• Click the .tgz link as per mention in screenshot.
• .
• Extract it within a separate directory using cmd tar -xvf kafka_file_with_extension.tgz.
• Create a directory named data and give the full permission to it sudo chmod ugo+rwx data.
• Also create directory inside the data directory named server and zookeeper and grant same permissions.
• Now configure the zookeeper and kafka properties.
• Now configure server.properties file update log.dirs=/home/kafka/data/server with newly created directory named server inside data directory also mentioned in the screenshot.
• .
• Now configure the zookeeper.properties file update dataDir=/home/kafka/data/zookeeper with newly created directory zookeeper inside the data directory also mentioned below screenshot.
• .
• In this below screenshots have all .sh files for kafka, zookeeper etc.
• .
• Now first run the zookeeper using command inside the bin directory ./zookeeper-server-start.sh ../config/zookeeper.properties.
• And then run the kafka using command inside the bin directory ./kafka-server-start.sh ../config/server.properties.
• Done.

[DevShivmohan]

Example steps to implement CA, truststore and keystore for apache kafka/zookeeper SSL/TLS security

• 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐞 𝐂𝐀 openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650.
• 𝐂𝐫𝐞𝐚𝐭𝐞 𝐓𝐫𝐮𝐬𝐭𝐬𝐭𝐨𝐫𝐞 keytool -keystore kafka.zookeeper.truststore.jks -alias ca-cert -import -file ca-cert.
• 𝐂𝐫𝐞𝐚𝐭𝐞 𝐊𝐞𝐲𝐬𝐭𝐨𝐫𝐞 keytool -keystore kafka.zookeeper.keystore.jks -alias zookeeper -validity 3650 -genkey -keyalg RSA -ext SAN=dns:localhost.
• 𝐂𝐫𝐞𝐚𝐭𝐞 𝐜𝐞𝐫𝐭𝐢𝐟𝐢𝐜𝐚𝐭𝐞 𝐬𝐢𝐠𝐧𝐢𝐧𝐠 𝐫𝐞𝐪𝐮𝐞𝐬𝐭 (𝐂𝐒𝐑) keytool -keystore kafka.zookeeper.keystore.jks -alias zookeeper -certreq -file ca-request-zookeeper.
• 𝐒𝐢𝐠𝐧 𝐭𝐡𝐞 𝐂𝐒𝐑 openssl x509 -req -CA ca-cert -CAkey ca-key -in ca-request-zookeeper -out ca-signed-zookeeper -days 3650 -CAcreateserial.
• 𝐈𝐦𝐩𝐨𝐫𝐭 𝐭𝐡𝐞 𝐂𝐀 𝐢𝐧𝐭𝐨 𝐊𝐞𝐲𝐬𝐭𝐨𝐫𝐞 keytool -keystore kafka.zookeeper.keystore.jks -alias ca-cert -import -file ca-cert
• 𝐈𝐦𝐩𝐨𝐫𝐭 𝐭𝐡𝐞 𝐬𝐢𝐠𝐧𝐞𝐝 𝐜𝐞𝐫𝐭𝐢𝐟𝐢𝐜𝐚𝐭𝐞 𝐢𝐧𝐭𝐨 𝐊𝐞𝐲𝐬𝐭𝐨𝐫𝐞 keytool -keystore kafka.zookeeper.keystore.jks -alias zookeeper -import -file ca-signed-zookeeper.
• Now integrate all generated certificates inside `/path//kafka/config/server.properties`** file by open the file on nano editor and configure it like as screenshot given below, replace the certificate file names as per created with name.
• .
• And the start first zookeeper by typing the command ./zookeeper-server-start.sh ../config/zookeeper.properties inside the bin directory as shown below screenshot.
• .
• Now run the apache kafka server by typing the command kafka-server-start.sh ../config/server.properties inside the bin directory as shown below screenshot.
• .

If there is occur any erros during run apache-kafka then try to delete all logs of zookeeper and kafka logs like below screenshot .

• First check log directory of kafka using this command sudo nano ./config/server.properties and check log.dirs value as per mention in the below screenshot.
• .
• And then goto the directory as per my system server logs directory is /home/kafka/data/server so delete everything inside the server directory by typing the command sudo rm -r *.
• And then check zookeeper data directory by typing the command sudo nano ./config/zookeeper.properties and check the value of dataDir as per given in screenshot.
• As per my system dataDir is /home/kafka/data/zookeeper so need to goto the directory by typing the command cd /home/kafka/data/zookeeper.
• And inside the directory may have exists something like version-2 directory, need also goto the directory version-2 and delete everything inside the version-2 directory by typing the command sudo rm -r *.
• Now run the zookeeper and then run the kafka-server.
• Done

[DevShivmohan]

In Spring-boot configuration with SSL/TLS apache kafka

• Generate all CA [Certificate Authority] files as per mention in this generate certificates link.
• And the paste the file paths of that files in application.properties as shown in given screenshot below.
• .
• Also take reference from application.properties -

  
  kafka.topic=normal-message
  spring.kafka.bootstrap.servers=localhost:9002
  spring.kafka.consumer.group-id=tbp
  server.port=9090

ssl.protocol = TLS spring.kafka.bootstrap-servers=localhost:9093 spring.kafka.properties.security.protocol=TLS spring.kafka.properties.ssl.truststore.location=file:/home/kafka/kafka_2.12-3.5.1/ssl/kafka.zookeeper.truststore.jks spring.kafka.properties.ssl.truststore.password=123456 spring.kafka.properties.ssl.keystore.location=file:/home/kafka/kafka_2.12-3.5.1/ssl/kafka.zookeeper.keystore.jks spring.kafka.properties.ssl.keystore.password=123456 spring.kafka.properties.ssl.key.password=123456

spring.kafka.producer.bootstrap-servers=localhost:9093 spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer spring.kafka.consumer.bootstrap-servers=localhost:9093 spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer