The Kubernetes Installer for Oracle Cloud Infrastructure provides a Terraform-based Kubernetes installation for Oracle Cloud Infrastructure. It consists of a set of Terraform modules and an example base configuration that is used to provision and configure the resources needed to run a highly available and configurable Kubernetes cluster on Oracle Cloud Infrastructure (OCI).
Terraform is used to provision the cloud infrastructure and any required local resources for the Kubernetes cluster including:
ssh_private_key
and ssh_public_key_openssh
input variablesca_cert
, ca_key
, etc.Terraform uses cloud-init scripts to handle the instance-level configuration for instances in the Control Plane to configure:
The Terraform scripts also accept a number of other input variables to choose instance shapes (including GPU) and how they are placed across the availability domain (ADs), etc. If your requirements extend beyond the base configuration, the modules can be used to form your own customized configuration.
~/.terraformrc
that specifies the path to the OCI provider:
providers {
oci = "<path_to_provider_binary>/terraform-provider-oci"
}
The OCI Cloud Controller Manager (CCM) and Volume Provisioner (VP) enables Kubernetes to dynamically provision OCI resources such as Load Balancers and Block Volumes as a part of pod and service creation. In order to facilitate this, OCI credentials and OCID information are automatically stored in the cluster as a Kubernetes Secret.
By default, the credentials of the user creating the cluster is used. However, in some cases, it makes sense to use a more restricted set of credentials whose policies are limited to a particular set of resources within the compartment.
To Terraform separate IAM users, groups, and policy resources, run the terraform plan
and terraform apply
commands from the identity
directory and set the appropriate input variables relating to your custom users, fingerprints, and key paths.
Create a terraform.tfvars file in the project root that specifies your configuration.
# start from the included example
$ cp terraform.example.tfvars terraform.tfvars
Initialize Terraform:
$ terraform init
View what Terraform plans do before actually doing it:
$ terraform plan
Use Terraform to Provision resources and stand-up k8s cluster on OCI:
$ terraform apply
The Kubernetes cluster will be running after the configuration is applied successfully and the cloud-init scripts have been given time to finish asynchronously. Typically, this takes around 5 minutes after terraform apply
and will vary depending on the overall configuration, instance counts, and shapes.
A working kubeconfig can be found in the ./generated
folder or generated on the fly using the kubeconfig
Terraform output variable.
Your network access settings determine whether your cluster is accessible from the outside. See Accessing the Cluster for more details.
If you've chosen to configure a public cluster, you can do a quick and automated verification of your cluster from
your local machine by running the cluster-check.sh
located in the scripts
directory. Note that this script requires your KUBECONFIG environment variable to be set (above), and SSH and HTTPs access to be open to etcd and worker nodes.
To temporarily open access SSH and HTTPs access for cluster-check.sh
, add the following to your terraform.tfvars
file:
# warning: 0.0.0.0/0 is wide open. remember to undo this.
etcd_ssh_ingress = "0.0.0.0/0"
master_ssh_ingress = "0.0.0.0/0"
worker_ssh_ingress = "0.0.0.0/0"
master_https_ingress = "0.0.0.0/0"
worker_nodeport_ingress = "0.0.0.0/0"
$ scripts/cluster-check.sh
[cluster-check.sh] Running some basic checks on Kubernetes cluster....
[cluster-check.sh] Checking ssh connectivity to each node...
[cluster-check.sh] Checking whether instance bootstrap has completed on each node...
[cluster-check.sh] Checking Flannel's etcd key from each node...
[cluster-check.sh] Checking whether expected system services are running on each node...
[cluster-check.sh] Checking status of /healthz endpoint at each k8s master node...
[cluster-check.sh] Checking status of /healthz endpoint at the LB...
[cluster-check.sh] Running 'kubectl get nodes' a number of times through the master LB...
The Kubernetes cluster is up and appears to be healthy.
Kubernetes master is running at https://129.146.22.175:443
KubeDNS is running at https://129.146.22.175:443/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://129.146.22.175:443/ui
Check out the example application deployment for a walk through of deploying a simple application that leverages both the Cloud Controller Manager and Flexvolume Driver plugins.
Check out the example cluster operations for details on how to use Terraform to scale, upgrade, replace, or delete your cluster.
--type=LoadBalancer
is an out-of-band change to Terraform. As a result, the cluster's VCN will not be able to be destroyed until all services of type LoadBalancer
have been deleted using kubectl
or the OCI Console.kubectl
or the OCI Console.Tests run automatically on every commit to the main branch. Additionally, the tests should be run against any pull-request before it is merged.
See Testing for details.
This project is open source. Oracle appreciates any contributions that are made by the open source community.
See Contributing for details.