Replace Portainer Setup with Minikube

[sebastian-sommerfeld-io]

Issue

• Controlling other Docker Compose Stacks with Portainer seems unstable.
• E.g. krokidile is always missing

Todo

• [x] Test Minikube setup in Vagrantbox
  • [x] Write ADR about Minikube vs Portainer (learning kubernetes and ArgoCD is part of the decision)
• [x] Setup Minikube for my workstations (directly, no VM to avoid all these port forwardings)
  • [x] Think about setting up and starting with Ansible or in another separate way (maybe a shell script with a bash alias or just a script without an alias next to configs/components/homelab/src/main/docker-stacks-cli.sh)
    • [x] Install with ansible
      • [x] Add minikube-cli alias
  • [x] Control minikube from a script or create some bash aliases for the most common commands
• [x] Remove Docker Stack GitOps control from Portainer (the Newman part)
  • [x] Remove the duplicated simple Portainer config
• [x] Use Portainer for monitoring and inspection purposes only
• [x] Deploy krokidile into Minikube with Helm
• [x] Deploy a Portainer instance into minikube with helm (admin-chart/portainer or similar)
  • [x] Somehow mount a file for the parameter --admin-password-file /tmp/portainer_passwords into the container (maybe with a config map or better a secret)
  • [x] Update Ditaa and header docs
• [x] Ops-Stack should remain unchanged and should not run inside Minikube
• [ ] Automate deployments so they react to changes automatically (through gitops). Scaling should be done automatically as well as updating a version. Move from helm to ArgoCD for this. Just deploy ArgoCD with helm as part of the admin-charts
  • [ ] Deploy ArgoCD into Minikube first
  • [ ] Deploy krokidike and source2adoc-website with ArgoCD
    • [ ] Expose krokidile and source2adoc-website through ingress
    • [ ] Think about DNS resolution (see https://chatgpt.com/share/ce90b437-475a-40fe-961f-bfb798e79e75)

Additional Todos

• [x] Refactor workflows to use semantic-release similar to krokidile
• [x] Add https://helm.sh/docs/helm/helm_lint to pre-commit and pipeline
  • [x] Exclude helm charts from yamllint
• [ ] Cause some traffic with gatlin to check if my deployments really run stable and respond to all requests
  • [ ] Run inside minikube? Or from the outside?
  • [ ] Place into src/test/minikube? Or is this a dedicated helm chart? Maybe placing outside if test dir is better because this feature is not used by a pipeline
• [x] Prepare inline docs for source2adoc
• [ ] PoC: Validate the Setup with Inspec. Minikiube setup the way I want? All services running?
• [ ] Setup Dependabot for Helm Charts if possible

Follow Up Todo

• [ ] Minikube should expose Prometheus metrics
• [ ] Think about a monitoring stack like Prometheus + Grafana + Loki to monitor Minikube. Start as a separate Docker Compose Stack to ensure Monitoring and Logging data is not lost, when Minikube crashes

Follow Up Poc

• [ ] Place a minishift setup (or better code ready containers - see comment below) next to minikube and deploy everything into minishift exactly the same way
  • [ ] Use a cli similar to minikube-cli (or maybe even use the same cli to decide which service to start)
  • [ ] Dynamically have kubectl and helm work with minikube or minishift (depending on which environment is started)
  • [ ] Add some sort of ADR or similar to document why I try out both. Maybe include in the scripts header docs.
• [ ] PoC: Test the minikube setup and its deployments on a RasPi node?

[sebastian-sommerfeld-io]

Comparison of Minikube and Minishift

Minikube and Minishift are both tools designed to simplify the process of running container orchestration platforms locally for development and testing purposes. While they share some similarities, they are tailored for different platforms and use cases.

Similarities

1. Local Development Environment:

   • Both Minikube and Minishift allow developers to run a single-node cluster locally on their machine for development and testing.

2. Cross-Platform Support:

   • Both tools support multiple operating systems, including Windows, macOS, and Linux.

3. Virtualization:

   • They both rely on virtual machines to create the cluster environment. They can use different VM drivers such as VirtualBox, KVM, Hyper-V, etc.

4. CLI Tools:

   • Minikube and Minishift provide command-line interfaces to manage the lifecycle of the local cluster, including starting, stopping, and configuring the cluster.

Differences

1. Underlying Platform:

   • Minikube: Primarily used for running a local Kubernetes cluster.
   • Minishift: Designed to run a local OpenShift cluster, which is a Kubernetes distribution with additional enterprise features.

2. Installation and Setup:

   • Minikube: Requires less initial setup and configuration, making it quicker to get started with a basic Kubernetes cluster.
   • Minishift: Involves setting up additional components specific to OpenShift, which might take more time and configuration.

3. Resource Requirements:

   • Minikube: Generally lighter in terms of resource usage since it focuses on Kubernetes without the additional overhead of OpenShift features.
   • Minishift: Requires more resources because it includes OpenShift-specific components like the web console, source-to-image builds, and more.

4. Use Case:

   • Minikube: Ideal for developers who need a straightforward, lightweight Kubernetes environment for testing and development.
   • Minishift: Best for developers and teams who are developing applications specifically for OpenShift or need to test features unique to OpenShift.

5. Add-ons and Integrations:

   • Minikube: Supports Kubernetes add-ons but does not include OpenShift-specific features like the OpenShift web console, integrated Jenkins, and OpenShift-specific routing.
   • Minishift: Comes with OpenShift-specific add-ons and features out of the box.

Which to Use and When

• Use Minikube if:

  • You are developing applications for Kubernetes.
  • You need a lightweight, quick-to-setup environment.
  • You are learning Kubernetes and do not need OpenShift-specific features.

• Use Minishift if:

  • You are developing applications specifically for OpenShift.
  • You need to test or develop using OpenShift-specific features and workflows.
  • Your organization uses OpenShift in production, and you want to ensure compatibility with the production environment.

Interaction with Helm, kubectl, and ArgoCD

Both Minikube and Minishift provide the ability to interact with your clusters using common Kubernetes tools such as Helm, kubectl, and ArgoCD. However, there are some nuances:

1. Helm:

   • Minikube: Fully supports Helm for managing Kubernetes applications. You can install and manage Helm charts just as you would on any Kubernetes cluster.
   • Minishift: Since OpenShift is Kubernetes-based, you can also use Helm with Minishift. However, some Helm charts may require modifications to work with OpenShift’s security constraints and policies.

2. kubectl:

   • Minikube: Directly supports kubectl for interacting with the Kubernetes API server.
   • Minishift: While you can use kubectl with OpenShift, it is more common to use the oc CLI, which is a superset of kubectl with additional OpenShift-specific commands and features.

3. ArgoCD:

   • Minikube: Fully compatible with ArgoCD for GitOps workflows, allowing you to deploy and manage applications via declarative GitOps practices.
   • Minishift: Also supports ArgoCD, but you might need to adjust certain configurations to align with OpenShift’s specific requirements and policies.

Unique CLI Tools

1. Minikube:

   • minikube CLI: Used to manage the lifecycle of the Minikube cluster. Commands include minikube start, minikube stop, minikube status, etc.

2. Minishift:

   • minishift CLI: Used to manage the lifecycle of the Minishift cluster. Commands include minishift start, minishift stop, minishift status, etc.
   • oc CLI: The OpenShift command-line interface, which extends kubectl with OpenShift-specific commands and features. Commands include oc new-app, oc status, oc project, etc.

Summary

• Helm: Works with both Minikube and Minishift, but some charts may need adjustments for OpenShift.

• kubectl: Native support for Minikube; works with Minishift, but the oc CLI is preferred.

• ArgoCD: Compatible with both, with potential configuration adjustments needed for OpenShift.

• Unique to Minikube: minikube CLI.

• Unique to Minishift: minishift CLI and oc CLI.

Using these tools effectively depends on your development environment and the specific requirements of your applications. Minikube offers simplicity for standard Kubernetes applications, while Minishift provides the additional capabilities required for developing and testing OpenShift-specific applications.

Installing Minikube and Minishift on the same machine

You can install both Minikube and Minishift on the same machine at the same time. Since you do not need to run both simultaneously, you can shut down one while powering up the other. This allows you to switch between a local Kubernetes cluster and a local OpenShift cluster as needed.

Steps to Manage Both on the Same Machine

1. Install Minikube:

   • Follow the installation instructions for Minikube on your operating system.
   • Start Minikube: minikube start
   • Stop Minikube when switching to Minishift: minikube stop

2. Install Minishift:

   • Follow the installation instructions for Minishift on your operating system.
   • Start Minishift: minishift start
   • Stop Minishift when switching to Minikube: minishift stop

Example Workflow

1. To use Minikube:

   minishift stop
   minikube start

2. To use Minishift:

   minikube stop
   minishift start

Considerations

• Ensure that your system has enough resources (CPU, RAM, disk space) to handle the virtual machines created by both Minikube and Minishift.
• Keep track of the versions of Kubernetes and OpenShift you are running to avoid conflicts or unexpected behavior.
• Be mindful of network configurations and any port forwarding settings that might interfere with each other.

By managing the start and stop of each tool, you can effectively use both Minikube and Minishift on the same machine without conflicts.

[sebastian-sommerfeld-io]

🚫 Minishift is no longer under active development

Minishift is no longer under active development. The project was officially deprecated in favor of CodeReady Containers (CRC), which provides a similar local OpenShift environment but with better performance and support for newer versions of OpenShift.

Key Points

• Minishift: Deprecated and not under active development. It supports older versions of OpenShift (typically up to OpenShift 3.11).
• CodeReady Containers (CRC): Actively maintained and recommended for running a local OpenShift 4.x cluster.

Recommendations

• For Local OpenShift Development: Use CodeReady Containers (CRC) instead of Minishift.
  • CRC provides a more streamlined and efficient way to run OpenShift 4.x on your local machine.
  • It comes with improved performance and better alignment with the latest OpenShift features and updates.

Transition from Minishift to CRC

If you have been using Minishift and need to switch to CRC, here are the basic steps:

1. Download CRC:

   • Get the latest version of CRC from the official Red Hat website.

2. Install CRC:

   • Follow the installation instructions provided on the CRC download page.

3. Start CRC:

   crc setup
   crc start

4. Stop CRC:

   crc stop

5. Usage:

   • CRC provides a crc command-line interface similar to Minishift.

By transitioning to CRC, you will ensure you are using a tool that is actively supported and aligned with the latest OpenShift developments.