NOTE: This project is still WIP
The Helm repository contains the charts and templates to provision the components into a given Kubernetes cluster. The charts are utilizing charts from Stackable organization which provides a wide selection of operators to work with technologies: Hadoop HDFS, Trino, Hbase, Zookeeper, etc. Stackable's operators provide a convenient and packaged solution for configuring and organization the technology stack within the range of technologies they support.
As GBIF current services lives outside of the Kubernetes ecosystem, the setup currently relies on a hard coded approach for exposing ports locally within GBIF infrastructure to focus on migrating the infrastructure for storage and processing first. By using Kubernetes, it allows us to utilize Stackable's operators and pre made images and later, a more smooth migration of other components and services if desired.
The way HDFS is setup in this project is very node dependent to rely on local assigned disks and ports to ensure fix locations for the pods and data.
Current components deployed via Helm charts:
It is possible to use the helmfile.yaml to install the whole cluster. BEWARE if you chooses to uninstall or reinstall the common-operator, the CRDs will be removed and thereby all the deployed components resulting in potential dataloss. The option currently kept for testing purposes.
In order to work with Helm charts you need the following tools:
The following plug-in is required:
In order for Helm to be able to communicate with the Kubernetes cluster, the local kube-context need to be configured.
In this project we are using semver 2.0 for the versioning of the helm charts. The charts can be released to GBIF helm repository by running the command:
helm cm-push charts/<chart_to_release> https://repository.gbif.org/repository/gbif-helm-release/ --context-path=/repository/gbif-helm-release --username=<your_nexus_username> --password=<your_nexus_password>
After a successful push the new version of the chart should be ready for use.
NOTE: Currently all charts are released to the snapshot repository as the first version is not ready.
Releasing to the snapshot repository is similar to previous command but pointing to the another repository:
helm cm-push charts/<chart_to_release> https://repository.gbif.org/repository/gbif-helm-snapshot/ --context-path=/repository/gbif-helm-snapshot --username=<your_nexus_username> --password=<your_nexus_password>
In GBIF we have adopted Helmfile to manage the state of our cluster contained in state files. The state files are stored in a separate repository to protect sensitive information.
The helm charts can still be deployed through the traditional way following these steps:
helm repo add gbif-charts-release https://repository.gbif.org/repository/gbif-helm-release/
helm dependency update charts/<chart_to_deploy>
helm upgrade --install <my_release> charts/<chart_to_deploy> -n <namespace_to_deploy_to> --set <key_1>=<value_1> --set <key_2>=<value_2>
Feel free to create an issue within this repository.
Describe the scenario the bug was spotted in and attach the bug label to the issue.
Describe the feature you are request together with an use-case scenario, use the enchantment label for the issue.