Local development on Kubernetes can be a hassle since the resources required to run a full Kubernetes cluster can be quite heavy.
In this post, we will explore running a single node Kubernetes cluster for your local development needs using Docker Compose.
Kubernetes Distribution
We will be focusing on 2 different Kubernetes distributions: K3s and RKE2.
K3s is a Rancher's lightweight, fully compliant Kubernetes distribution, that is packaged into a single binary. It is most commonly used for edge computing or IoT use cases, and there are projects such as k3d which is a lightweight wrapper to run K3s in docker.
RKE2 is Rancher's next-generation Kubernetes distribution, combining RKE1's close alignment with upstream Kubernetes and K3s's deployment model for ease-of-operations. It also comes with FIPS 140-2 compliance.
K3s in Docker
k3d provides a simple solution to create k3s clusters for local development, but we are looking for an even simpler solution that only uses Docker and Docker Compose.
You only need a docker-compose.yml file:
version: '3.7'
services:
server:
image: rancher/k3s:v1.24.0-rc1-k3s1-amd64
networks:
- default
command: server
tmpfs:
- /run
- /var/run
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
privileged: true
restart: always
environment:
# - K3S_TOKEN=${K3S_PASSWORD_1} # Only required if we are running more than 1 node
- K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
- K3S_KUBECONFIG_MODE=666
volumes:
- k3s-server:/var/lib/rancher/k3s
# This is just so that we get the kubeconfig file out
- ./k3s_data/kubeconfig:/output
- ./k3s_data/docker_images:/var/lib/rancher/k3s/agent/images
expose:
- "6443" # Kubernetes API Server
- "80" # Ingress controller port 80
- "443" # Ingress controller port 443
ports:
- 6443:6443
volumes:
k3s-server: {}
networks:
default:
ipam:
driver: default
config:
- subnet: "172.98.0.0/16" # Self-defined subnet on local machine
Create the k3s_data directory and run the docker compose up command:
mkdir -p k3s_data/kubeconfig
docker compose up -d
All that is left is to alias your kubectl to use your K3s kubeconfig, and you can start interacting with your K3s Kubernetes cluster!
alias k='kubectl --kubeconfig '"${PWD}"'/k3s_data/kubeconfig/kubeconfig.yaml'
k get pods -A
# NAMESPACE NAME READY STATUS RESTARTS AGE
# kube-system local-path-provisioner-7b7dc8d6f5-q5ldx 1/1 Running 0 4m23s
# kube-system coredns-b96499967-kw4gt 1/1 Running 0 4m23s
# kube-system metrics-server-668d979685-cvxmv 1/1 Running 0 4m23s
# kube-system helm-install-traefik-crd-wrjz6 0/1 Completed 0 4m24s
# kube-system helm-install-traefik-smxtr 0/1 Completed 1 4m24s
# kube-system svclb-traefik-pxlgs 2/2 Running 0 3m51s
# kube-system traefik-7cd4fcff68-lbg4m 1/1 Running 0 3m51s
RKE2 in Docker
RKE2 is slightly more complicated as there isn't a readily available container image. We will have to build our own container image and publish it to our container registry of choice.
The Dockerfile can be found here. Alternatively, I have already built the container image and you can copy my image.
Once again, the docker-compose.yml file:
version: '3.7'
services:
server:
image: sachua/rke2-test:v1.27.1-rke2r1
networks:
- default
command: server
tmpfs:
- /run
- /var/run
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
privileged: true
restart: always
environment:
# - RKE2_TOKEN=${RKE2_PASSWORD_1} # Only required if we are running more than 1 node
- RKE2_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
- RKE2_KUBECONFIG_MODE=666
volumes:
- rke2-server:/var/lib/rancher/rke2
# This is just so that we get the kubeconfig file out
- ./rke2_data/kubeconfig:/output
- ./rke2_data/docker_images:/var/lib/rancher/rke2/agent/images
expose:
- "6443" # Kubernetes API Server
- "80" # Ingress controller port 80
- "443" # Ingress controller port 443
ports:
- 6443:6443
volumes:
rke2-server: {}
networks:
default:
ipam:
driver: default
config:
- subnet: "172.98.0.0/16"
Create the rke2_data directory and run the docker compose up command:
mkdir -p rke2_data/kubeconfig
docker compose up -d
Finally, alias your kubectl to use your RKE2 kubeconfig, and you can start interacting with your RKE2 Kubernetes cluster!
RKE2 might take longer to load as it has more initial set-up tasks than K3s
Local development on Kubernetes can be a hassle since the resources required to run a full Kubernetes cluster can be quite heavy.
In this post, we will explore running a single node Kubernetes cluster for your local development needs using Docker Compose.
Kubernetes Distribution
We will be focusing on 2 different Kubernetes distributions:
K3s
andRKE2
.K3s is a Rancher's lightweight, fully compliant Kubernetes distribution, that is packaged into a single binary. It is most commonly used for edge computing or IoT use cases, and there are projects such as k3d which is a lightweight wrapper to run K3s in docker.
RKE2 is Rancher's next-generation Kubernetes distribution, combining RKE1's close alignment with upstream Kubernetes and K3s's deployment model for ease-of-operations. It also comes with FIPS 140-2 compliance.
K3s in Docker
k3d provides a simple solution to create k3s clusters for local development, but we are looking for an even simpler solution that only uses Docker and Docker Compose.
You only need a
docker-compose.yml
file:Create the
k3s_data
directory and run thedocker compose up
command:All that is left is to alias your kubectl to use your K3s kubeconfig, and you can start interacting with your K3s Kubernetes cluster!
RKE2 in Docker
RKE2 is slightly more complicated as there isn't a readily available container image. We will have to build our own container image and publish it to our container registry of choice.
The Dockerfile can be found here. Alternatively, I have already built the container image and you can copy my image.
Once again, the
docker-compose.yml
file:Create the
rke2_data
directory and run thedocker compose up
command:Finally, alias your kubectl to use your RKE2 kubeconfig, and you can start interacting with your RKE2 Kubernetes cluster! RKE2 might take longer to load as it has more initial set-up tasks than K3s