// INSTRUCTION: Please remove all comments that start INSTRUCTION prior to commit. Most comments should be removed, although not the copyright.
// INSTRUCTION: The copyright statement must appear at the top of the file
//
// Copyright (c) 2018 IBM Corporation and others.
// Licensed under Creative Commons Attribution-NoDerivatives
// 4.0 International (CC BY-ND 4.0)
// https://creativecommons.org/licenses/by-nd/4.0/
//
// Contributors:
// IBM Corporation
//
:projectid: caching-microservices-hazelcast
:page-layout: guide
:page-duration: 25 minutes
:page-releasedate: 2019-02-15
:page-description: Explore how to use caching in microservices within Kubernetes environment.
:page-tags: ['Hazelcast', 'Caching', 'microservices', 'Kubernetes', 'Containers','Spring Boot' , 'Minikube']
:page-permalink: /guides/{projectid}
:page-related-guides: ['docker', 'kubernetes-intro']
:common-includes: https://raw.githubusercontent.com/OpenLiberty/guides-common/master
:source-highlighter: prettify
:page-seo-title: Caching in microservices with Hazelcast Tutorial
:page-seo-description: How to use Hazelcast with microservices
= Caching Microservices with Hazelcast in Kubernetes
[.hidden]
NOTE: This repository contains the guide documentation source. To view the guide in published form, view it on the https://openliberty.io/guides/{projectid}.html[Open Liberty website].
Use Hazelcast Caching in Open Liberty and Spring Boot based Microservices and deploy to Kubernetes
:minikube-ip: 192.168.99.100
:kube: Kubernetes
:hashtag: #
:win: WINDOWS
:mac: MAC
:linux: LINUX
:hazelcast: Hazelcast
// =================================================================================================
// What you'll learn
// =================================================================================================
== What you'll learn
You will learn how to use Hazelcast distributed caching with Spring Boot, bundle with openliberty and
deploy to a local {kube} cluster.
You will then create a Kubernetes Service which load balance between containers and
verify that you can share data between Microservices.
The microservice you will deploy is called hazelcast-caching
. The hazelcast-caching
microservice simply
helps you put a data and read it back. As Kubernetes Service will send the request to different pod each time
you initiate the request, the data will be served by shared hazelcast cluster between hazelcast-caching
pods.
You will use a local single-node {kube} cluster. However, you can deploy this application on any kubernetes distributions
such as IBM Cloud Private.
== What is {hazelcast}?
Hazelcast is an open source In-Memory Data Grid (IMDG). It provides elastically scalable distributed In-Memory computing,
widely recognized as the fastest and most scalable approach to application performance.
Hazelcast is designed to scale up to hundreds and thousands of members.
Simply add new members and they will automatically discover the cluster
and will linearly increase both memory and processing capacity
== Why Spring Boot?
Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that you can "just run".
To learn more about Spring Boot.
http://spring.io/projects/spring-boot
// =================================================================================================
// Prerequisites
// =================================================================================================
include::{common-includes}/kube-prereq.adoc[]
// =================================================================================================
// Getting Started
// =================================================================================================
include::{common-includes}/gitclone.adoc[]
// no "try what you'll build" section in this guide since it would be too long due to all setup the user will have to do.
// =================================================================================================
// Staring and preparing your cluster for deployment
// =================================================================================================
include::{common-includes}/kube-start.adoc[]
// =================================================================================================
// Use Hazelcast Caching in Spring Boot Application
// =================================================================================================
== Use Hazelcast Caching in Spring Boot Application
The microservice in the start
directory has in-process cache which is not accessible between microservices.
Firstly, you need to configure Hazelcast to be used in Spring Boot Application.
replace Application.java
with the following java class.
[source, java]
include::finish/src/main/java/io/openliberty/guides/hazelcast/Application.java[tags=**;]
This configuration will tell Spring Boot what kind of configuration is needed when each hazelcast member is started.
Now it is time to use Hazelcast in the hazelcast-caching microservice.
if you check the source code of CommandController.java
, you would notice that ConcurrentHashMap
is used
for backing key value store. In order to make this cache distributed, replace CommandController.java
with Hazelcast version.
[source, java]
include::finish/src/main/java/io/openliberty/guides/hazelcast/CommandController.java[tags=**;]
You would notice that getting hazelcast member is Autowired
by Spring Boot and made available to the CommandController
and there is no change in /put
and /get
endpoints.
// =================================================================================================
// Building and containerizing the microservices
// =================================================================================================
== Building and containerizing microservice
The first step of deploying to {kube} is to build your microservices and containerize them with Docker.
The starting Java project, which you can find in the start
directory, is a spring boot based microservice.
It is called hazelcast-caching which has two simple endpoints, /put
and /get
. This guide uses boost-maven-plugin
to bundle spring boot app and openliberty together and build docker image for the deployment.
Boost plugin does not require you to create your own Dockerfile but instead opionated docker image.
If you wanna learn more about boost maven plugin
, you can visit the project website.
https://github.com/OpenLiberty/boost/tree/master/boost-maven
If you want to create your own Dockerfile then you can always use dockerfile-maven plugin.
please refer to https://openliberty.io/guides/kubernetes-intro.html#starting-and-preparing-your-cluster-for-deployment
To build hazelcast-caching
microservice, run the following command in the start
folder.
This will build the docker image and push it to docker registry provided by minikube
[role='command']
mvn package
Execute following command to see available docker images
[role='command']
docker images
Verify that the hazelcast-caching:latest
is listed among them, for example:
[system]#{win} | {mac}#
[source, role="no_copy"]
REPOSITORY TAG IMAGE ID CREATED SIZE
hazelcast-caching latest ee0fb6a53f68 11 seconds ago 595MB
991f5e8961a5 13 seconds ago 595MB
open-liberty springBoot2 1da856d69fac 3 weeks ago 546MB
k8s.gcr.io/kube-proxy-amd64 v1.10.0 bfc21aadc7d3 11 months ago 97MB
k8s.gcr.io/kube-apiserver-amd64 v1.10.0 af20925d51a3 11 months ago 225MB
k8s.gcr.io/kube-scheduler-amd64 v1.10.0 704ba848e69a 11 months ago 50.4MB
k8s.gcr.io/kube-controller-manager-amd64 v1.10.0 ad86dbed1555 11 months ago 148MB
k8s.gcr.io/etcd-amd64 3.1.12 52920ad46f5b 11 months ago 193MB
k8s.gcr.io/kube-addon-manager v8.6 9c16409588eb 12 months ago 78.4MB
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64 1.14.8 c2ce1ffb51ed 13 months ago 41MB
k8s.gcr.io/k8s-dns-sidecar-amd64 1.14.8 6f7f2dc7fab5 13 months ago 42.2MB
k8s.gcr.io/k8s-dns-kube-dns-amd64 1.14.8 80cc5ea4b547 13 months ago 50.5MB
k8s.gcr.io/pause-amd64 3.1 da86e6ba6ca1 14 months ago 742kB
k8s.gcr.io/kubernetes-dashboard-amd64 v1.8.1 e94d2f21bc0c 14 months ago 121MB
gcr.io/k8s-minikube/storage-provisioner v1.8.0 4689081edb10 15 months ago 80.8MB
gcr.io/k8s-minikube/storage-provisioner v1.8.1 4689081edb10 15 months ago 80.8MB
k8s.gcr.io/pause-amd64 3.0 99e59f495ffa 2 years ago 747kB
----
[system]#*{linux}*#
[source, role="no_copy"]
----
REPOSITORY TAG IMAGE ID CREATED SIZE
hazelcast-caching latest ee0fb6a53f68 11 seconds ago 595MB
991f5e8961a5 13 seconds ago 595MB
open-liberty springBoot2 1da856d69fac 3 weeks ago 546MB
k8s.gcr.io/kube-proxy-amd64 v1.10.0 bfc21aadc7d3 11 months ago 97MB
k8s.gcr.io/kube-apiserver-amd64 v1.10.0 af20925d51a3 11 months ago 225MB
k8s.gcr.io/kube-scheduler-amd64 v1.10.0 704ba848e69a 11 months ago 50.4MB
k8s.gcr.io/kube-controller-manager-amd64 v1.10.0 ad86dbed1555 11 months ago 148MB
k8s.gcr.io/etcd-amd64 3.1.12 52920ad46f5b 11 months ago 193MB
k8s.gcr.io/kube-addon-manager v8.6 9c16409588eb 12 months ago 78.4MB
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64 1.14.8 c2ce1ffb51ed 13 months ago 41MB
k8s.gcr.io/k8s-dns-sidecar-amd64 1.14.8 6f7f2dc7fab5 13 months ago 42.2MB
k8s.gcr.io/k8s-dns-kube-dns-amd64 1.14.8 80cc5ea4b547 13 months ago 50.5MB
k8s.gcr.io/pause-amd64 3.1 da86e6ba6ca1 14 months ago 742kB
k8s.gcr.io/kubernetes-dashboard-amd64 v1.8.1 e94d2f21bc0c 14 months ago 121MB
gcr.io/k8s-minikube/storage-provisioner v1.8.0 4689081edb10 15 months ago 80.8MB
gcr.io/k8s-minikube/storage-provisioner v1.8.1 4689081edb10 15 months ago 80.8MB
k8s.gcr.io/pause-amd64 3.0 99e59f495ffa 2 years ago 747kB
----
****
If you don't see the `hazelcast-caching:latest` image, then check the Maven build log for any potential errors.
In addition, if you are using Minikube, make sure your Docker CLI is configured to use Minikube's Docker daemon and not your host's as described in the previous section.
// =================================================================================================
// Deploying the microservices
// =================================================================================================
== Deploying the microservices
Now that your Docker images are built, deploy them using a Kubernetes resource definition.
To deploy the `hazelcast-caching` microservice, first create the `kubernetes.yaml` file in the `start` directory:
[source, yaml]
----
include::finish/kubernetes.yaml[tags=**;]
----
This file defines two {kube} resources: one statefulset and one service.
StatefulSet is preferred solution for Hazelcast because it enables controlled scale out/in of your microservices
for easy data distribution. To learn more about StatefulSet, you can visit Kubernetes documentation
https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/
By default, we create 2 replicas of `hazelcast-caching` microservice behind the `hazelcast-caching-service` which forwards
requests to one of the pods available in the kubernetes cluster.
`MY_POD_NAME` is an environment variable made available to the pods so that each microservice knows which pod they are in.
This is going to be used in this guide in order to show which pod is responding to the http request.
As a second step, we will create rbac.yaml
[source, yaml]
----
include::finish/rbac.yaml[tags=**;]
----
Role Based Access Controller(RBAC) configuration is used to give access to Kubernetes Master API from pods which runs
microservices. Hazelcast requires a read access to autodiscover other hazelcast members and form hazelcast cluster.
Run the following commands to deploy the resources as defined in kubernetes.yaml and rbac.yaml in the specified order:
[role='command']
```
kubectl apply -f rbac.yaml
```
[role='command']
```
kubectl apply -f kubernetes.yaml
```
run the following command to check the status of your pods:
[role='command']
```
kubectl get pods
```
You'll see an output similar to the following if all the pods are healthy and running:
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
hazelcast-caching-statefulset-0 1/1 Running 0 20m
hazelcast-caching-statefulset-1 1/1 Running 0 20m
----
You should also check if hazelcast cluster is formed by checking one of the pod's log file:
```
kubectl logs hazelcast-caching-statefulset-1
```
[source, role="no_copy"]
----
Members {size:2, ver:2} [
Member [172.17.0.4]:5701 - 71009ef7-ee18-45f0-8a8f-e9321931e9ce this
Member [172.17.0.5]:5701 - 99222e16-93e5-4453-ac9e-cdf3e80069c6
]
----
Next you will make requests to your services.
****
[system]#*{win} | {mac}*#
The default hostname for Docker Desktop is `localhost`.
[system]#*{linux}*#
The default hostname for minikube is {minikube-ip}. Otherwise it can be found using the `minikube ip` command.
****
Open your favorite terminal and send http get requests in a loop via `curl` command.
This request asks for the value of `key=1` and prints value and which pod has replied to the request.
You might need to wait up to 30 sec before microservices accepts traffic.
```
while true; do curl 192.168.99.100:31000/get?key=1;echo; sleep 2; done
```
You should see an output like below. The `value` is null because we have not put any data yet.
`podname` shows that which kubernetes pod replied to the request.
[source, role="no_copy"]
----
{"value":null,"podName":"hazelcast-caching-statefulset-1"}
{"value":null,"podName":"hazelcast-caching-statefulset-1"}
{"value":null,"podName":"hazelcast-caching-statefulset-0"}
{"value":null,"podName":"hazelcast-caching-statefulset-1"}
{"value":null,"podName":"hazelcast-caching-statefulset-0"}
----
Break the current loop and put some data into hazelcast-caching microservice.
[role='command']
```
curl "http://192.168.99.100:31000/put?key=1&value=hazelcast_springboot_openliberty"
```
Although request has been executed by one specific kubernetes pod, by querying the data in a loop will show data the data
is actually shared by multiple pods. To see this, execute following command again.
[role='command']
```
while true; do curl 192.168.99.100:31000/get?key=1;echo; sleep 2; done
```
As you can see both pods are returning the same data.
[source, role="no_copy"]
----
{"value":"hazelcast_springboot_openliberty","podName":"hazelcast-caching-statefulset-1"}
{"value":"hazelcast_springboot_openliberty","podName":"hazelcast-caching-statefulset-0"}
{"value":"hazelcast_springboot_openliberty","podName":"hazelcast-caching-statefulset-1"}
{"value":"hazelcast_springboot_openliberty","podName":"hazelcast-caching-statefulset-0"}
{"value":"hazelcast_springboot_openliberty","podName":"hazelcast-caching-statefulset-1"}
----
// =================================================================================================
// Scaling with Hazelcast
// =================================================================================================
== Scaling with Hazelcast
Scale the cluster with one more pod and see that you still retrieve the shared data.
[role='command']
```
kubectl scale statefulset hazelcast-caching-statefulset --replicas=3
```
Run following command to see the latest status of the pods
[role='command']
```
kubectl get pods
```
As you can see, a new pod `hazelcast-caching-statefulset-2` has joined to the cluster.
[source, role="no_copy"]
----
NAME READY STATUS RESTARTS AGE
hazelcast-caching-statefulset-0 1/1 Running 0 8m
hazelcast-caching-statefulset-1 1/1 Running 0 8m
hazelcast-caching-statefulset-2 1/1 Running 0 31s
----
Run the following command again to see the output
[role='command']
```
while true; do curl 192.168.99.100:31000/get?key=1;echo; sleep 2; done
```
As you can see, `hazelcast-caching-statefulset-2` is returning correct data.
[source, role="no_copy"]
----
{"value":"hazelcast_springboot_openliberty","podName":"hazelcast-caching-statefulset-1"}
{"value":"hazelcast_springboot_openliberty","podName":"hazelcast-caching-statefulset-2"}
{"value":"hazelcast_springboot_openliberty","podName":"hazelcast-caching-statefulset-0"}
----
// =================================================================================================
// Testing microservices that are running on {kube}
// =================================================================================================
== Testing microservices that are running on {kube}
[role="code_command hotspot", subs="quotes"]
----
#Create a `HazelcastCachingIT` class.#
`src/test/java/it/io/openliberty/guides/hazelcast/HazelcastCachingIT.java`
----
HazelcastCachingIT.java
[source, Java, linenums, indent=0, role="code_column"]
----
include::finish/src/test/java/it/io/openliberty/guides/hazelcast/HazelcastCachingIT.java[tags=**;!copyright;!comment]
----
The `testHazelcastCache` test makes sure that the `/put` endpoint is handled by one pod
and `/get` methods returns the same data from the other kubernetes pod.
It first puts a key/value pair to `hazelcast-caching` microservice and keeps podname in the `firstpod`
variable. In the second part, tests submits multiple `/get` requests until to see that podname is different then the pod
which initially handled `/put` request.
In order to run integration tests, you must have a running `hazelcast-caching` microservices
in minikube environment. As you have gone through all previous steps, you already have it.
The default properties defined in the pom.xml are:
[cols="15, 100", options="header"]
|===
| *Property* | *Description*
| [hotspot=35]`cluster.ip` | IP or hostname for your cluster, `{minikube-ip}` by default, which is appropriate when using Minikube.
| [hotspot=36]`cluster.port` | Port of the {kube} Service wrapping the `hazelcast-caching` pods, `hazelcast-caching-service` by default.
|===
Navigate back to the `start` directory.
****
[system]#*{linux} | {mac}*#
Run the integration tests against a cluster running at the default Minikube IP address:
[role='command']
```
mvn verify
```
[system]#*{win}*#
Run the integration tests against a cluster running with a hostname of `localhost`:
[role='command']
```
mvn verify -Dcluster.ip=localhost
```
Run the integration tests against an external IP address and port,
you can use `cluster.ip` and `cluster.port` based kubernetes service on your kubernetes cluster.
[role='command']
```
mvn verify -Dcluster.ip=192.168.99.100 -Dcluster.port=31000
```
If the tests pass, you'll see a similar output to the following:
```
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running it.io.openliberty.guides.hazelcast.HazelcastCachingIT
14:52:17.881 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Created GET request for "http://192.168.99.100:31000/put?key=1&value=hazelcast-springboot-openliberty"
14:52:17.916 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Setting request Accept header to [application/json, application/*+json]
14:52:17.989 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - GET request for "http://192.168.99.100:31000/put?key=1&value=hazelcast-springboot-openliberty" resulted in 200 (OK)
14:52:17.991 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Reading [class io.openliberty.guides.hazelcast.CommandResponse] as "application/json;charset=UTF-8" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@48e08f49]
14:52:18.007 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Created GET request for "http://192.168.99.100:31000/get?key=1"
14:52:18.007 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Setting request Accept header to [application/json, application/*+json]
14:52:18.048 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - GET request for "http://192.168.99.100:31000/get?key=1" resulted in 200 (OK)
14:52:18.048 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Reading [class io.openliberty.guides.hazelcast.CommandResponse] as "application/json;charset=UTF-8" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@48e08f49]
14:52:19.051 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Created GET request for "http://192.168.99.100:31000/get?key=1"
14:52:19.052 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Setting request Accept header to [application/json, application/*+json]
14:52:19.060 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - GET request for "http://192.168.99.100:31000/get?key=1" resulted in 200 (OK)
14:52:19.060 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Reading [class io.openliberty.guides.hazelcast.CommandResponse] as "application/json;charset=UTF-8" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@48e08f49]
14:52:20.069 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Created GET request for "http://192.168.99.100:31000/get?key=1"
14:52:20.069 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Setting request Accept header to [application/json, application/*+json]
14:52:20.122 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - GET request for "http://192.168.99.100:31000/get?key=1" resulted in 200 (OK)
14:52:20.122 [Time-limited test] DEBUG org.springframework.web.client.RestTemplate - Reading [class io.openliberty.guides.hazelcast.CommandResponse] as "application/json;charset=UTF-8" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@48e08f49]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.714 s - in it.io.openliberty.guides.hazelcast.HazelcastCachingIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.3.RELEASE:stop (post-integration-test) @ hazelcast-caching ---
[INFO]
[INFO] --- maven-failsafe-plugin:2.21.0:verify (default) @ hazelcast-caching ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.470 s
[INFO] Finished at: 2019-03-04T14:52:20-05:00
[INFO] ------------------------------------------------------------------------
```
// =================================================================================================
// Tear Down
// =================================================================================================
== Tearing down the environment
When you no longer need your deployed microservices, you can delete all {kube} resources by running the `kubectl delete` command:
You might need to wait up to 30 seconds as stateful sets kills pods one at a time.
```
kubectl delete -f kubernetes.yaml
```
include::{common-includes}/kube-minikube-teardown.adoc[]
// =================================================================================================
// finish
// =================================================================================================
== Great work! You're done!
You have just created a Spring Boot application, bundled it with openliberty and deployed to {kube}.
You then added {hazelcast} caching to the `hazelcast-caching`,tested with a simple curl command.
You also scaled out the microservices and saw that data is shared between microservices
As a last step, you ran integration tests against `hazelcast-caching` that was deployed in a {kube} cluster.
// Include the below from the guides-common repo to tell users how they can contribute to the guide
include::{common-includes}/finish.adoc[]
// DO NO CREATE ANYMORE SECTIONS AT THIS POINT
// Related guides will be added in automatically here if you included them in ":page-related-guides"