gothub
commented
6 years ago Tests were run on the cluster on 20180402.
The test workload consisted of 1000 quality report generation requests for a metadata document stored on disk, local to the quality engine. One worker pod contained the metadig-controller, which received the report requests and from 1-15 metadig-worker pods created the quality reports. Four test runs were conducted, each run with a different number of worker pods.
Each of the two nodes (ubuntu VMS) in the k8s cluster has 12 CPU cores and 16GB of memory.
Test results
| run # | # worker pods | Elapsed time (minutes) | Ave. worker elapsed (seconds) | memory usage (GB) docker 1 | memory usage (GB) docker2 | load ave docker1 | load ave docker2 |
|---|---|---|---|---|---|---|---|
| 1 | 1 | 21 | 1 | 1.2 | 4.7 | .49 | 1.1 |
| 2 | 5 | 4 | 1 | 7.3 | 6.6 | 3.6 | 2.7 |
| 3 | 10 | 2 | 1 | 11.2 | 12.8 | 4.2 | 6.0 |
| 4 | 15 | 5 | 4 | 15.3 | 15.4 | 79.1 | 59.2 |
Interpretation of results
The cluster performance appears to begin to degrade with only 15 workers, when the total cluster memory available approaches exhaustion.
The total cluster CPU usage was never close to maximum usage.
Potential ways to improve performance
Decrease memory footprint of metadig-worker
Starting with OpenJDK 8, the docker images available from https://hub.docker.com/_/openjdk/ have some level of optimization for running inside a container. Here is a description from Docker Hub for the java options recommended for use inside containers:
Inside Linux containers, recent versions of OpenJDK 8 can correctly detect container-limited number of CPU cores by default. To enable the detection of container-limited amount of RAM the following options can be used:
java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap ...These arguments might also be used in conjunction with -Xms, which specifies the initial Java heap size
Note that the above reports statistics for this performance test did not use any of these java command line arguments, but another
round of tests will be run using these arguments.
Decrease Docker image size for metadig-worker
The current docker image for metadig-worker is 173MB in size, and uses the openjdk:8-jre-alpine docker tagged image. This image seems to load quickly, as metadig-worker pods are created in a few seconds. The default docker image openjdk:8, is about 800MB and takes almost a minute to load.
The docker image might be made even smaller using other image variants, however this might introduce other problems as non-standard libraries are used for some of the smaller image variants.
mbjones
In order to check the feasibility of using Kubernetes (k8s) for use with metadig engine, representative workloads will be run on the k8s cluster (docker-ucsb-1.test.dataone.org, docker-ucsb-2).