ifdef::env-github[] :status: :tip-caption: :bulb: :note-caption: :information_source: :important-caption: :heavy_exclamation_mark: :caution-caption: :fire: :warning-caption: :warning: :table-caption!: :namespace: postgresql-operator endif::[]
:toc: :toc-placement!:
== Database Operator
ifdef::status[] .Project health image:https://circleci.com/gh/dev4devs-com/postgresql-operator.svg?style=svg[Build Status (CircleCI), link=https://circleci.com/gh/dev4devs-com/postgresql-operator] image:https://img.shields.io/:license-Apache2-blue.svg[License (License), link=http://www.apache.org/licenses/LICENSE-2.0] image:https://coveralls.io/repos/github/dev4devs-com/postgresql-operator/badge.svg?bVerifyingranch=master[Coverage Status (Coveralls), link=https://coveralls.io/github/dev4devs-com/postgresql-operator?branch=master] image:https://goreportcard.com/badge/github.com/dev4devs-com/postgresql-operator[Go Report Card (Go Report Card), link=https://goreportcard.com/report/github.com/dev4devs-com/postgresql-operator] endif::[]
:toc: toc::[]
== Overview
A very flexible and customizable https://commons.openshift.org/sig/OpenshiftOperators.html[Operator] in Go developed using the https://github.com/operator-framework[ Operator Framework] to package, install, configure and manage a Database database.
== Prerequisites
The following prerequisites are just required if you would like to install it as standalone (without OLM).
|=== |https://golang.org/doc/install[Install Golang] v1.13+ |Access to a Kubernetes v1.16.0+ cluster. |https://github.com/golang/go/wiki/SettingGOPATH[Ensure the $GOPATH environment variable is set] |https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl[Install kubectl] |===
NOTE: If you haven't a access to a Kubernetes v1.16.0+ cluster. It is recommend the usage of https://kubernetes.io/docs/setup/learning-environment/minikube/[Minikube] for local tests and development purposes. You may would like to check the blog posts https://dev4devs.com/2019/10/19/how-to-getting-started-with-kubernetes/[How to getting started with Kubernetes?].
The following prerequisite is just required if you would like to contribute with.
|=== |https://github.com/operator-framework/operator-sdk#quick-start[Install Operator-SDK] |===
== Getting Started
NOTE: The following steps will allow you use this project as standalone (without OLM).
=== Cloning the repository
By the following commands you will create a local directory and clone this project.
=== Installing as standalone
Use the following command to install the Operator and Database
NOTE: To install you need be logged in as a user with cluster privileges like the system:admin user.
=== Verifying
To verify that the installation was successful completed you can check the Database Status field of link:deploy/crds/postgresql.dev4devs.com_databases_crd.yaml[Database CRD] in the cluster. The expected result when all was installed with success is OK.
$ kubectl describe Database -n postgresql-operator | grep Status
...
Status:
Database Status: OK
...Now is time to check if all the resources are available as expected, running kubectl get all -o wide -n postgresql-operator one should see a result similar to the one below
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
pod/database-5485c7f8d4-6khcw 1/1 Running 1 14m 172.17.0.9 localhost
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/database ClusterIP 172.30.87.113
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/database 1 1 1 1 14m database centos/postgresql-96-centos7 cr=database,owner=postgresqloperator deployment.apps/postgresql-operator 1 1 1 1 14m postgresql-operator quay.io/dev4devs-com/postgresql-operator:master name=postgresql-operator
=== Uninstalling
Use the following command to delete the Operator, Database and Backup Service if enabled.
NOTE: To uninstall you need be logged in as a user with cluster privileges like the system:admin user.
== OperatorHub.io
This operator can be installed by the OperatorHub.io. Note that application image is deployed at https://quay.io/repository/dev4devs-com/postgresql-operator
Tip: If for some reason you have problems interacting with quay.io, go to "https://quay.io/user/
== Configuration and Options
=== Using configMap to share required values
By the specs in link:deploy/crds/postgresql.dev4devs.com_v1alpha1_database_cr.yamlDatabase CR you are able to customize the setup for this operator. Note that by the spec configMapName you are able to inform the name of a configMapName which has the keys and values which the Database should use in its required env vars.
If you inform only the name of the configMap at configMapName, then it will look for the values stored with the same keys required for each image env var used for its database version (databaseName, databasePassword, databaseUser). However, you are able to customize the keys as well by using the optional specs; configMapDatabaseName, configMapDatabasePassword, configMapDatabaseUser. This way, this operator will be able to look for the values stored in some config with keys which are not the ones used to create the environment variables used in the database deployment.
=== Changing the operator namespace
By using the command make install as it is, the default namespace will be postgresql-operator, defined in the link:./Makefile[Makefile] file, it will be created and the operator installed in this namespace. You are able to install the operator in another namespace if you wish, however, you need to set up its roles (RBAC) in order to apply them on the namespace where the operator will be installed. The namespace name needs to be changed in the link:./deploy/rolebinding.yaml[Cluster Role Binding](/deploy/rolebinding.yaml) file. Note, that you also need to change the namespace in the link:./Makefile[Makefile] in order to use the command make install with a different namespace.
=== Configuring the Backup Service
==== Backup
===== Install The backup service is implemented by using https://github.com/integr8ly/backup-container-image[integr8ly/backup-container-image]. It will do the database backup to be restored later in case of failures. Following the steps to enable it.
+
IMPORTANT: You can add the name of the secret which is created already in the cluster.
+
IMPORTANT: You need to create the bucket yourself
+
. Run the command make install-backup in the same namespace where the Database is installed in order to apply the CronJob which will do this process.
NOTE: To install you need be logged in as a user with cluster privileges like the system:admin user.
===== Verifying
To verify if the backup has been successfully created you can run the following command in the namespace where the operator is installed.
To check the jobs executed you can run the command kubectl get jobs -n postgresql-operator in the namespace where the operator is installed as in the following example.
NOTE: In the above example the schedule was made to run this job each minute (*/1 * * * *)
To check the logs and troubleshooting you can run the command kubectl logs $podName -f -n postgresql-operator in the namespace where the operator is installed as the following example.
==== Restore
Following the steps required to be performed a database restore based in the backup service.
. Install the Database by following the steps in <gunzip -c filename.gz | psql dbname
== Architecture
This operator is cluster-scoped. For further information see the https://github.com/operator-framework/operator-sdk/blob/master/doc/user-guide.md#operator-scope[Operator Scope] section in the Operator Framework documentation. Also, check its roles in link:./deploy/[Deploy] directory.
NOTE: The operator and database will be installed in the namespace {namespace} which will be created by this project.
=== CRD Definitions
|=== | CustomResourceDefinition | Description | link:deploy/crds/postgresql.dev4devs.com_databases_crd.yaml[Database] | Packages, manages, installs and configures the Database on the cluster. | link:deploy/crds/postgresql.dev4devs.com_backups_crd.yaml[Backup] | Packages, manages, installs and configures the CronJob to do the backup using the image https://github.com/integr8ly/backup-container-image[backup-container-image] |===
=== Resources managed by each CRD Controller
link:./pkg/controller/postgresql/controller.go[Database Database]
|=== | Resource | Description | link:./pkg/resource/deployments.go[deployments.go] | Define the Deployment resource of Database. (E.g container and resources definitions) | link:./pkg/resource/pvs.go[pvs.go] | Define the PersistentVolumeClaim resource used by its Database. | link:./pkg/resource/services.go[services.go] | Define the Service resource of Database. |===
link:./pkg/controller/backup/controller.go[Backup]
|=== | Resource | Description | link:./pkg/resource/cronjobs.go[cronjobs.go] | Define the CronJob resources in order to do the Backup. | link:./pkg/resource/secrets.go[secrets.go] | Define the database and AWS secrets resources created. |===
== Administration
=== Status Definition per Types
link:./pkg/apis/postgresql-operator/v1alpha1/postgresql_types.go[Database]
|===
| Status | Description
| databaseStatus | For this status is expected the value OK which means that all required objects are created.
| deploymentStatus | Deployment Status from ks8 API (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#deploymentstatus-v1-apps[appsv1.DeploymentStatus]).
| serviceStatus | Deployment Status from ks8 API (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#servicestatus-v1-core[v1core.ServiceStatus]).
| PersistentVolumeClaimStatus | PersistentVolumeClaim Status from ks8 API (persistentvolumeclaimstatus[v1core.PersistentVolumeClaimStatus])
|===
link:./pkg/apis/postgresql-operator/v1alpha1/backup_types.go[Backup]
|===
| Status | Description
| backupStatus | Should show OK when everything is created successfully.
| cronJobName | Name of cronJob resource created by it.
| cronJobStatus | CronJob Status from ks8 API (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#cronjobstatus-v1beta1-batch[v1beta1.CronJobStatus]).
| dbSecretName | Name of database secret resource created in order to allow the https://github.com/integr8ly/backup-container-image[integr8ly/backup-container-image] connect to the database .
| awsSecretName | Name of AWS S3 bucket secret resource used in order to allow the https://github.com/integr8ly/backup-container-image[integr8ly/backup-container-image] connect to AWS to send the backup .
| awsCredentialsSecretNamespace | Namespace where the backup image will looking for the of the Aws Secret used.
| encryptKeySecretName | Name of the EncryptKey used.
| encryptKeySecretNamespace | Namespace where the backup image will looking for the of the EncryptKey used.
| hasEncryptionKey | Expected true when it was configured to use an EncryptnKey secret
| isDatabasePodFound | The value expected here is true which shows that the database pod was found.
| isDatabaseServiceFound | The value expected here is true which shows that the database service was found.
|===
== Development
=== Local Setup
Run the following command to setup this project locally.
NOTE: It is using go modules to manage dependencies.
=== Running the operator locally
The following command will install the operator in the cluster and run the changes performed locally without the need to publish a dev tag. In this way, you can verify your code in the development environment.
IMPORTANT: The local changes are applied when the command operator-sdk up local --namespace={namespace} is executed then it is not a hot deploy and to get the latest changes you need re-run the command.
=== Checking Database
By the following commands you are able to connect to the Database. It is possible to check it through the OpenShift UI in the Database's pod terminal.
psql -U postgres
\c
\dt
=== Debugging
Follow the below steps to debug the project in some IDEs.
NOTE: The code needs to be compiled/built first.
==== IntelliJ IDEA / GoLand
Then, debug the project from the IDE by using the default setup of Go Remote option.
==== Visual Studio Code
debug the project using the following Visual Studio Code launch config.
== Makefile command reference
=== Application Commands
|===
| Command | Description
| make install | Creates the {namespace} namespace, application CRDS, cluster role and service account. Installs the operator and DB
| make uninstall | Uninstalls the operator and DB. Deletes the {namespace}`` namespace, application CRDS, cluster role and service account. i.e. all configuration applied bymake install |make install-backup| Installs the backup Service in the operator's namespace |make uninstall-backup` | Uninstalls the backup Service from the operator's namespace.
|===
=== Local Development
|===
| make code-run-local | Runs the operator locally for development purposes.
| make setup-debug | Sets up environment for debugging proposes.
| make vet | Examines source code and reports suspicious constructs using https://golang.org/cmd/vet/[vet].
| make fmt | Formats code using https://golang.org/cmd/gofmt/[gofmt].
| make gen | It will automatically generated/update the files by using the operator-sdk based on the CR status and spec definitions.
| make dev | It will tun the dev commands to check, fix and generated/update the files.
|===
=== CI
|===
| make image-build-master | Used by CI to build operator image from master branch and add :master tag.
| make image-push-master | Used by CI to push the master image to https://quay.io/repository/dev4devs-com/postgresql-operator[quay.io registry].
| make image-build-release | Used by CI to build operator image from a tagged commit and add :<version> and latest tag.
| make image-push-release | Used by CI to push the release and latest image to https://quay.io/repository/dev4devs-com/postgresql-operator[quay.io registry].
|===
=== Tests
|===
| make test | Runs test suite
| make integration-cover | Run coverage check
| make compile-e2e | Compile image for tests
| make test-e2e | Run locally e2e tests (Required have cluster installed locally)
|===
NOTE: The link:./Makefile[Makefile] is implemented with tasks which you should use to work with.
== Publishing images
Images are automatically built and pushed to our https://quay.io/repository/dev4devs-com/postgresql-operator[image repository] in the following cases:
master tag is published.<operator-version> and latest tags are published.If the image does not get built and pushed automatically the job may be re-run manually via the https://circleci.com/gh/dev4devs-com/postgresql-operator[CI dashboard].
== Release Tags
Following the steps
0.1.0NOTE: The image with the tag will be created and pushed to the https://quay.io/repository/dev4devs-com/postgresql-operator[postgresql-operator image hosting repository] by the CI.
WARNING: Do not use letters in the tag such as v. It will not work.
== Release OLM Catalogue
Use the following command.
NOTE: See here some examples link:./deploy/olm-catalog/olm-test[/deploy/olm-catalog/olm-test] which can be used to test it.
NOTE: You can use the command operator-sdk scorecard to check it locally. Update the link:./.osdk-scorecard.yaml[config file] with the latest changes
== Supportability
This operator was developed using the Kubernetes APIs in order to be compatible with OpenShift and Kubernetes.
== Contributing
All contributions are hugely appreciated. Please see our link:./CONTRIBUTING.md[Contribution Guide] for guidelines on how to open issues and pull requests. Please check out our link:./CODE_OF_CONDUCT.md[Code of Conduct] too.