Chapter 16: problem with test-em-all.bash and Kubernetes

[hjoly2003]

Hi, while running the code in chapter 16, I'm experiencing problems with the test-em-all.bash script when it is running in a Kubernetes cluster.

I was trying to reproduce what you did in the section "Deploying to Kubernetes for development and test". My version can be found on GitHub at the following URL: Chapter16_init. Since I have a Mac M1, I had to adapt your version of the project (See the "arm64" bullet in the README.md file). A detailed description of the steps that I did can be found in the section Deploying to Kubernetes for development and test of the same README.md file.

First, I started a cluster inspired by what you did in chapter 15, but without ingress and the metric-server (see Develop, build and deploy microservices on Apple silicon (ARM64) | Callista).

Then, for gradle to build successfully, I had to skip the unit tests since some tests were failing (see the "gradle build" bullet). Note that if I run gradle build outside of the minikube environment, they all pass.

After that, I was able to follow the rest of your instructions without any problem up to the test_em_all.sh script. When I start the script, it waits for ever for a response from the actuator health.

Start Tests: Ven 27 jan 2023 07:22:53 EST
HOST=192.168.67.2
PORT=30443
USE_K8S=true
SKIP_CB_TESTS=false
Wait for: curl -k https://192.168.67.2:30443/actuator/health... , retry #1 , retry #2 , retry #3 , retry #4 , retry #5 , retry #6 , retry #7 , retry #8

Note that I have the same problem if I use a plain minikube cluster (i.e. via minikube start).

I hope that in my GitHub repository, you'll find everything you need to diagnose the problem.

Finally, I'd like to thank you for your book. So far, it is the most complete reference on the subject and I would recommend it to everyone.

Thanks in advance

[JunChen22]

Is your review service running fine? I see this after you typed this command $ kubectl get pods --watch

review-67ccb9ffcc-ldlxd 0/1 Running 1 (5s ago) 3m21s

I had something messed up in mine and it won't run. I just type $ minikube delete to restart the whole process. And type$ kubectl describe NAME_OF_THE_POD and $ kubectl logs NAME_OF_THE_POD on the pod is not running. Something like $ kubectl describe review-67ccb9ffcc-ldlxd. The test won't run unless all services is up and running.

I didn't run your code, I just saw your README.MD.

[hjoly2003]

Hi, when I ran kubectl get pods -watch, I would eventually get the following:

NAME                                 READY   STATUS    RESTARTS      AGE
auth-server-76d8c786d4-jdvvw         1/1     Running   0             61s
config-server-f474799bb-dg2pr        1/1     Running   0             61s
gateway-84c4d8599f-b8mjr             1/1     Running   0             61s
mongodb-8594c8fc45-bwj28             1/1     Running   0             61s
mysql-7dbd64ff56-vqlnn               1/1     Running   0             61s
product-55dcb77f5f-4xjrm             1/1     Running   0             61s
product-composite-5bccc45877-bfbgg   1/1     Running   0             61s
rabbitmq-cd46cc675-5kr86             1/1     Running   0             61s
recommendation-776848d5b9-x9vkl      1/1     Running   0             61s
review-6f657fbbc6-hv8sp              1/1     Running   0             61s
zipkin-server-6d89bd9b89-vx7nz       1/1     Running   2 (49s ago)   61s

As you can see, all my services were started successfully. I confirmed that with the following kubectl wait --timeout=600s --for=condition=ready pod --all and it replied:

pod/auth-server-76d8c786d4-jdvvw condition met
pod/config-server-f474799bb-dg2pr condition met
pod/gateway-84c4d8599f-b8mjr condition met
pod/mongodb-8594c8fc45-bwj28 condition met
pod/mysql-7dbd64ff56-vqlnn condition met
pod/product-55dcb77f5f-4xjrm condition met
pod/product-composite-5bccc45877-bfbgg condition met
pod/rabbitmq-cd46cc675-5kr86 condition met
pod/recommendation-776848d5b9-x9vkl condition met
pod/review-6f657fbbc6-hv8sp condition met
pod/zipkin-server-6d89bd9b89-vx7nz condition met

Like I've said in my first email, the problem comes when I start the test-em-all.bash.

Thanks

[hjoly2003]

As an update, I realized that I had to include ingress and the metric-server to my cluster. Of course, I had! (removal of Eureka in chapter 16)!

I re-conducted my tests but, despite my expectations, it ended up with exactly the same problems.

By the way, I was able to fully reproduce the Kubernetes tutorial of chapter 15.

Thanks in advance,

[hjoly2003]

Hi, to ease the troubleshooting, I have included the logs of the different microservices once they were booted up. See all the *.log files under the root directory of my GitHub depot (see Chapter16_init).

Thanks in advance

[magnus-larsson]

After a discussion with @hjoly2003 over mail, we concluded that the incorrect IP address was used as the external address for the Minikube cluster. When using the Docker driver on an M1-based Mac, the external address is always 127.0.0.1 or localhost, i.e. the same as when using the Docker driver in the Win10/WSL 2 environment.

The book uses the Hyperkit driver on Intel-based Macs, and therefore the external address is provided by the command minikube ip. For further details, see page 484 in the PDF.