search

star3am / hashiqube

HashiQube - The Ultimate Hands on DevOps Lab running All the HashiCorp Products in a Github Codespace or a Docker Container using Vagrant or Docker Compose
https://hashiqube.com
Other
93 stars 41 forks source link
airflow ansible ansible-project boundary consul dbt devops gitlab hashicorp hashiqube jenkins minikube nomad packer terraform vagrant vault waypoint

readme

HashiQube - A DevOps Development Lab Using All the HashiCorp Products

HashiQube

Overview

Hashiqube is the Ultimate Hands on DevOps Lab in a Docker Container. It can run in a Github Codespace, Locally using Vagrant or Docker Compose, or as a Virtual Machine VM on AWS, GCP and Azure.

Hashiqube has a Docker daemon inside meaning, we can run containers inside Hashiqube using Kubernetes (Minikube) or Nomad or Docker run. It runs all Hashicorp products. Vault, Terraform, Nomad, Consul, Waypoint, Boundary, Vagrant, Packer and Sentinel.

It also runs a host of other popular Open Source DevOps/DevSecOps applications (Minikube, Ansible AWX Tower, Traefik etc.) showcasing how simple integration with Hashicorp products can result in tangible learnings and benefits for all its users.

Once Hashiqube is up an internet connection is no longer needed meaning sales pitches and demos for potential and existing customers are greatly aided.

Hashiqube has been created to help Engineers, Developers and anyone who wants to practise, learn or demo HashiCorp products to get started quickly with a local lab.

Please connect with me on LinkedIn (Riaan Nolan) or check out my Credly profile

Running Hashiqube

There are a few ways of running Hashiqube depending on your needs

Hashiqube on Github Codespaces

HashiQube on Github's Codespaces

Hashiqube Locally

Follow the instructions below to run Hashiqube locally on your laptop or PC.

Pre-requisites

:bulb: If you want to run Minikube and a workload like AWX Ansible Tower, or Airflow you need at least 8G RAM - If you want to run Gitlab as well, you need to give Docker resources at least 12G RAM

Installation Instructions

:clock3: Duration 15 - 30 minutes
:bulb: Docker is the Default and preferred way to run Hashiqube

Docker Compose

Docker compose is also supported! To bring up Hashiqube with Docker Compose do the following

Now run the following commands: docker compose exec hashiqube /bin/bash This will drop you into Hashiqube as root, now you can bring up the individual services, start with Basetools
bash hashiqube/basetools.sh This will install a bunch of dependencies
bash docker/docker.sh This will install the Docker Daemon inside
Hashiqube bash vault/vault.sh This will bring up HashiCorp Vault

And so on, you get the idea..

See: docker compose ls

NAME                STATUS              CONFIG FILES
hashiqube           running(1)          /Users/riaan/workspace/personal/hashiqube/docker-compose.yml

Documentation and Status

:loudspeaker: For Hashiqube Local Documentation and Status after you complete the steps above in Installation Instructions

:book: Documentation: http://localhost:3333
:vertical_traffic_light: Status of Integrations: http://localhost:3001

HashiQube Overview

HashiQube: A Jedi DevOps Lab Using All the HashiCorp Products

HashiQube Diagrams

In essence, Hashiqube is a Docker Container (by default) or a Virtual Machine.

Hashiqube has a Docker Daemon inside meaning we can run docker containers/pods inside Hashiqube using Kubernetes or Nomad or Docker run.

Hashiqube runs on your local laptop, PC or a cloud instance.

Hashiqube runs all the Hashicorp products and other popular Open Source tools, such as Minikube, Ansible AWX Tower, Prometheus and Grafana and many, many more, see: Hashiqube Integrations

You can also run Hashiqube on AWS, GCP and Azure. This is called Hashiqube Multi-Cloud, see: Multi-Cloud

Hashiqube is a Training / Development Lab for you to practise, learn or demo POC stuff with, it should not be run in production.

Hashiqube can be visualized with the diagrams below.

Draw.io Diagram

filename

Links

Hashiqube Integrations

HashiQube

Once the stack is up you will have a large number of services running and available on localhost
For Documentation please open http://localhost:3333 in your browser

Hashiqube Integrations

HashiQube's Purpose

Hashiqube has been created to enable anyone who is interested in secure automation pipelines the ability to run a suite of ‘best in class’ tools on their local machines at the cost of a small amount of system resources.

Hashiqube gives all interested parties the empowerment to deploy these tools in a way that covers multiple use cases effectively providing a ‘concept to completion’ test bed using open-source Hashicorp products.

The original use case was born from the desire to demystify DevSecOps utilising Terraform, Vault, Consul, Sentinel and Nomad as well as some other well known open source CI/CD tools by providing a ‘hands-on’ environment that demonstrates the value of secret and credential management in a standard software development pipeline.

Thanks to the flexibility of the Hashicorp products there is no need to wonder how to achieve the goals of bringing software to market in a more secure and timely fashion, just Vagrant up!

Supported Architectures

Name Docker Virtualbox Hyper-V
amd64 ✓ ✓ ✘
arm64 ✓ ✘ ✘
linux ✓ ✓ ✘
windows ✓ ✘ ✘
mac intel ✓ ✓ ✘
mac apple ✓ ✘ ✘

Supported HashiCorp product Versions

By default Hashiqube installs community editions of Hashicorp products, but you can also test and demo Enterprise versions. You can request a trial license from here https://developer.hashicorp.com/vault/tutorials/enterprise/hashicorp-enterprise-license#request-a-trial-license

When you get the license file, simply copy the license.hclic into the vault or corresponding product folder in the hashiqube directory and run:

vagrant up --provision-with basetools,vault,consul,nomad,boundary

Name Community Enterprise
Vault ✓ ✓
Consul ✓ ✓
Nomad ✓ ✓
Boundary ✓ ✓
Terraform ✓ ✘

tree -L 1 boundary consul nomad vault

boundary
├── README.md
├── boundary.sh
├── images
└── license.hclic
consul
├── README.md
├── consul.sh
├── images
└── license.hclic
nomad
├── README.md
├── images
├── license.hclic
├── nomad
└── nomad.sh
vault
├── README.md
├── images
├── license.hclic
└── vault.sh

Components

Hashiqube is made up of a number of components and some rely on each other like building blocks.

For example, you can run components separately as demonstrated below.

vagrant up --provision-with basetools
vagrant up --provision-with docker
vagrant up --provision-with docsify
vagrant up --provision-with vault
vagrant up --provision-with consul
vagrant up --provision-with nomad
vagrant up --provision-with minikube

Or one-shot as demonstrated below.

vagrant up --provision-with basetools,docker,minikube,postgresql,dbt,apache-airflow

Docker Desktop

Docker Desktop is an easy-to-install application for your Mac or Windows environment that enables you to build and share containerized applications and microservices. It's a graphical user interface for the docker service.

If you have HashiQube running, you won't see any containers but you will be able to open the application.

Docker Desktop

Now that docker is installed we need to ensure that the docker environment and settings are configured

Docker Desktop Resources

:bulb: You can quite happily run Hashiqube with 4G (minimum) or 8G (recommended) of RAM, but once you start running, Vault, Nomad, Consul, Waypoint, Boundary AND you want to run Minikube with let's say AWX Ansible Tower at the same time, the processes will start contending for resources and you will get weird errors. For that reason, the below screenshot is my setup with 12G of RAM. For reference, I use an M1 Macbook Air with 16G or RAM as my personal laptop. I frequently run all of HashiQube's services to test with, for example, I'll run Vault, Nomad, Consul, Waypoint, Boundary and Minikube with Grafana and Prometheus to test the monitoring of Vault, Nomad and Consul, for that reason 4G Ram would not be enough.

Docker Desktop Resources

Consul DNS

Local DNS via Consul
Add on our local Macbook a file /etc/resolver/consul with below contents

nameserver 10.9.99.10
port 8600

Now you can use DNS like nomad.service.consul:9999 vault.service.consul:9999 via Fabio Load Balancer

The HashiStack

Dimension Products
Applications Nomad
Nomad
Schedular and workload orchestrator to deploy and manage applications		 Waypoint
Waypoint
One workflow to build, deploy and release applications across platforms
Networking Consul
Consul
Service Mesh across any cloud and runtime platform
Security Boundary
Boundary
Secure remote access to applications and critical systems		 Vault
Vault
Secure management of secrets and sensitive data
Infrastructure Packer
Packer
Automated machine images from a single source configuration		 Vagrant
Vagrant
Single workflow to build and manage developer environments		 Terraform
Terraform
Infrastructure automation to provision and manage any cloud service

Other

Vagrant Basic Usage

Docker Basic Usage

Errors you might encounter

:bulb: If you see this error message

vagrant destroy

    hashiqube0: Are you sure you want to destroy the 'hashiqube0' VM? [y/N] y
There are errors in the configuration of this machine. Please fix
the following errors and try again:

shell provisioner:
* `path` for shell provisioner does not exist on the host system: /Users/riaan/workspace/personal/hashiqube/vault/vault.sh

Command docker ps 

CONTAINER ID IMAGE COMMAND CREATED STATUS  PORTS NAMES
1d835d757279   15f77507dce7   "/usr/sbin/init"   38 hours ago   Up 38 hours   0.0.0.0:1433->1433/tcp, 0.0.0.0:3000->3000/tcp, 0.0.0.0:3306->3306/tcp, 0.0.0.0:3333->3333/tcp, 0.0.0.0:4566->4566/tcp, 0.0.0.0:4646-4648->4646-4648/tcp, 0.0.0.0:5001-5002->5001-5002/tcp, 0.0.0.0:5432->5432/tcp, 0.0.0.0:5580->5580/tcp, 0.0.0.0:5601-5602->5601-5602/tcp, 0.0.0.0:7777->7777/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8043->8043/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8088->8088/tcp, 0.0.0.0:8181->8181/tcp, 0.0.0.0:8200-8201->8200-8201/tcp, 0.0.0.0:8300-8302->8300-8302/tcp, 0.0.0.0:8500-8502->8500-8502/tcp, 0.0.0.0:8888-8889->8888-8889/tcp, 0.0.0.0:9001-9002->9001-9002/tcp, 0.0.0.0:9011->9011/tcp, 0.0.0.0:9022->9022/tcp, 0.0.0.0:9090->9090/tcp, 0.0.0.0:9093->9093/tcp, 0.0.0.0:9200->9200/tcp, 0.0.0.0:9333->9333/tcp, 0.0.0.0:9701-9702->9701-9702/tcp, 0.0.0.0:9998-9999->9998-9999/tcp, 0.0.0.0:10888->10888/tcp, 0.0.0.0:11888->11888/tcp, 0.0.0.0:18080->18080/tcp, 0.0.0.0:18181->18181/tcp, 0.0.0.0:18888-18889->18888-18889/tcp, 0.0.0.0:19200->19200/tcp, 0.0.0.0:19701-19702->19701-19702/tcp, 0.0.0.0:28080->28080/tcp, 0.0.0.0:31506->31506/tcp, 0.0.0.0:32022->32022/tcp, 0.0.0.0:8600->8600/udp, 0.0.0.0:2255->22/tcp, 0.0.0.0:33389->389/tcp   hashiqube_hashiqube0_1689246032

Solution run docker stop 1d835d757279 

The IP address configured for the host-only network is not within the
allowed ranges. Please update the address used to be within the allowed
ranges and run the command again.

Address: 10.9.99.10
Ranges: 192.168.56.0/21

Valid ranges can be modified in the /etc/vbox/networks.conf file. For
more information including valid format see:

https://www.virtualbox.org/manual/ch06.html#network_hostonly

Please create the following file: /etc/vbox/networks.conf with the following contents

* 10.0.0.0/8 192.168.0.0/16
* 2001::/64

and re-run vagrant up --provision

Error response from daemon: cannot stop container: 6c0c8135620ff47efe12df417a0df0e57d7a81a7f7ca06d011323fbb52e573db: tried to kill container, but did not receive an exit event
Command vagrant destroy
Solution run vagrant destroy again 

    hashiqube0.service.consul: Are you sure you want to destroy the 'hashiqube0.service.consul' VM? [y/N] y
==> hashiqube0.service.consul: Stopping container...
A Docker command executed by Vagrant didn't complete successfully!
The command run along with the output from the command is shown
below.

Command: ["docker", "stop", "-t", "1", "6c0c8135620ff47efe12df417a0df0e57d7a81a7f7ca06d011323fbb52e573db", {:notify=>[:stdout, :stderr]}]

Stderr: Error response from daemon: cannot stop container: 6c0c8135620ff47efe12df417a0df0e57d7a81a7f7ca06d011323fbb52e573db: tried to kill container, but did not receive an exit event

Error The IP address configured for the host-only network is not within the allowed ranges. Please update the address used to be within the allowed ranges and run the command again.
Command vagrant up --provision
Solution Ensure the following contents are present in /etc/vbox/networks.conf 

* 10.0.0.0/8 192.168.0.0/16
* 2001::/64

Error Vagrant cannot forward the specified ports on this VM, since they would collide with some other application that is already listening on these ports. The forwarded port to 9200 is already in use on the host machine.

To fix this, modify your current project's Vagrantfile to use another port. For example, where '1234' would be replaced by a unique host port: 

config.vm.network :forwarded_port, guest: 9200, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this case, Vagrant was unable to. This is usually because the guest machine is in a state which doesn't allow modifying port forwarding. You could try 'vagrant reload' (the equivalent of running a halt followed by an up) so vagrant can attempt to auto-correct this upon booting. Be warned that any unsaved work might be lost.
Command vagrant up --provision
Info When I run vagrant up I get an error about a port collision, in this case, it is port 9200 - The reason is that I have an Elasticsearch container running on my local laptop, and since HashiQube also run Elasticsearch this port is taken.
Solution I stop the Elasticsearch docker container and I run vagrant up again, or I hash the line out in the Vagrantfile like the example below 

# config.vm.network "forwarded_port", guest: 9200, host: 9200 # elasticsearch

Support & Feedback

For suggestions, feedback and queries please branch or and submit a Pull Request or directly contact Riaan Nolan, creator of the HashiQube via Github - hashiqube

About Hashiqube

Hashiqube runs all the Hashicorp products and a host of other popular Open Source software that is heavily used in the industry.

Once you have done vagrant up --provision you will have access to Vault, Nomad, Consul, Boundary, Waypoint and this documentation page on your local computer.

In addition to the Core Hashicorp products, Hashiqube also runs a host of other popular Open Source integrations that are heavily used within the industry today.

Hashiqube Integrations

About Me

My name is Riaan Nolan and I was born in South Africa. I started out as a Web Developer in 2000 and from there progressed into Systems Administration, with a strong focus on Automation, Infrastructure and Configuration as Code.

I have worked for Multi-National companies in Portugal, Germany, China, South Africa, United States and Australia.

Please connect with me on LinkedIn or check out my Credly profile

My Hashicorp Badges

Contributors and Special mentions

A Very special mention to HashiQube's contributors, Thank You All for your help, suggestions and contributions no matter how small <3

Videos

Videos were made with asciinema

License

HashiQube is free and available for personal use under the terms of the BSL License.

Hashiqube Logo Automate all the things DevOps Lab Logo

google ads