Basic Steps/Tutorial On how to use the Operator?

[kuro337]

hey!

I was running a Kubernetes Cluster and I was exploring to migrate it from VM's to my Clusters - which are bare metal spun up using Kubeadm.

I was testing how to use this on a simple Kubernetes cluster with 2 worker nodes - just to see if this is manageable to use.

I tried following the docs - https://access.crunchydata.com/documentation/postgres-operator/latest/quickstart and https://access.crunchydata.com/documentation/postgres-operator/latest/tutorials/basic-setup/create-cluster

But after following all of the docs and scouring through the Github issues , I wasn't able to find anything relevant. I was wondering if there was a complete End to End Documentation that doesn't assume Dynamic Storage is available?

For folks testing it out and learning , typically they will not have access to Dynamic PV Provisioning - maybe there is relevant documentation but I was not able to find anything , would appreciate if anyone could point me towards it!

The complete steps I followed were:

# clone forked repo
git clone https://github.com/forked/postgres-operator-examples.git

# install the crds first
kubectl apply -k kustomize/install/namespace
kubectl apply --server-side -k kustomize/install/default
# success no errors

Then I am editing the operator file - to specify my Storage Classes created on Each Node

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: hippo
spec:
  image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-16.2-0
  postgresVersion: 16
  instances:
    - name: instance1
      dataVolumeClaimSpec:
        storageClassName: "pg-local-storage" # Added this line
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 4Gi
  backups:
    pgbackrest:
      image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.49-0
      repos:
        - name: repo1
          volume:
            volumeClaimSpec:
              storageClassName: "pg-local-storage" # Added this line
              accessModes:
                - "ReadWriteOnce"
              resources:
                requests:
                  storage: 4Gi

This is my bash script that I use to create the PV, PVC, and Storage Class before running the Operator

#!/bin/bash

# Creates a Storage Class in Envs with no Dynamic PV Provisioning
# Create dirs on each node beforehand
# sudo mkdir -p /mnt/data/postgres-data

NODE_NAMES=("worker1" "worker2" )
STORAGE_CLASS="pg-local-storage"
PVC_SIZE="4Gi"
STANDARD_PATH="/mnt/data/postgres-data"

# Define storage class
kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: $STORAGE_CLASS
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF

for NODE in "${NODE_NAMES[@]}"; do
  PV_NAME="postgres-pv-${NODE//./-}" 
  echo "Creating PV: $PV_NAME for Node: $NODE at $STANDARD_PATH"

  kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: $PV_NAME
spec:
  capacity:
    storage: $PVC_SIZE
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: $STORAGE_CLASS
  local:
    path: $STANDARD_PATH
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - "$NODE"
EOF
done

# Create a PersistentVolumeClaim
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: $PVC_SIZE
  storageClassName: $STORAGE_CLASS
EOF

Then I launch the Postgres using Operator

kubectl apply -k kustomize/postgres

But this way seems to not work - if anyone could even point me to a better way to do it or documentation would highly appreciate it!

My outputs after running commands for Reference with errors

user@controlnode:~/scripts/pv$ kubectl -n postgres-operator get pods \
  --selector=postgres-operator.crunchydata.com/cluster=hippo,postgres-operator.crunchydata.com/instance
NAME                     READY   STATUS    RESTARTS   AGE
hippo-instance1-6429-0   4/4     Running   0          8m19s
user@controlnode:~/scripts/pv$ kubectl -n postgres-operator describe postgresclusters.postgres-operator.crunchydata.com hippo
Name:         hippo
Namespace:    postgres-operator
Labels:       <none>
Annotations:  <none>
API Version:  postgres-operator.crunchydata.com/v1beta1
Kind:         PostgresCluster
Metadata:
  Creation Timestamp:  2024-03-03T22:39:33Z
  Finalizers:
    postgres-operator.crunchydata.com/finalizer
  Generation:        1
  Resource Version:  3298321
  UID:               1980fb9b-31f1-43a6-87bc-084c04430ace
Spec:
  Backups:
    Pgbackrest:
      Image:  registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.49-0
      Repos:
        Name:  repo1
        Volume:
          Volume Claim Spec:
            Access Modes:
              ReadWriteOnce
            Resources:
              Requests:
                Storage:         4Gi
            Storage Class Name:  pg-local-storage
  Image:                         registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-16.2-0
  Instances:
    Data Volume Claim Spec:
      Access Modes:
        ReadWriteOnce
      Resources:
        Requests:
          Storage:         4Gi
      Storage Class Name:  pg-local-storage
    Name:                  instance1
    Replicas:              1
  Port:                    5432
  Postgres Version:        16
Status:
  Conditions:
    Last Transition Time:  2024-03-03T22:39:34Z
    Message:               pgBackRest dedicated repository host is not ready
    Observed Generation:   1
    Reason:                RepoHostNotReady
    Status:                False
    Type:                  PGBackRestRepoHostReady
    Last Transition Time:  2024-03-03T22:39:34Z
    Message:               pgBackRest replica create repo is not ready for backups
    Observed Generation:   1
    Reason:                StanzaNotCreated
    Status:                False
    Type:                  PGBackRestReplicaRepoReady
    Last Transition Time:  2024-03-03T22:39:34Z
    Message:               pgBackRest replica creation is not currently possible
    Observed Generation:   1
    Reason:                RepoBackupNotComplete
    Status:                False
    Type:                  PGBackRestReplicaCreate
  Database Revision:       f886c77f4
  Instances:
    Name:              instance1
    Ready Replicas:    1
    Replicas:          1
    Updated Replicas:  1
  Monitoring:
    Exporter Configuration:  559c4c97d6
  Observed Generation:       1
  Patroni:
    System Identifier:  7342270977082974279
  Pgbackrest:
    Repo Host:
      API Version:  apps/v1
      Kind:         StatefulSet
      Ready:        false
    Repos:
      Name:            repo1
      Stanza Created:  false
  Proxy:
    Pg Bouncer:
      Postgres Revision:  5c9966f6bc
  Users Revision:         5569cddf7b
Events:
  Type     Reason                 Age                  From                        Message
  ----     ------                 ----                 ----                        -------
  Normal   RepoHostCreated        8m25s                postgrescluster-controller  created pgBackRest repository host StatefulSet/hippo-repo-host
  Warning  UnableToCreateStanzas  24s (x6 over 5m35s)  postgrescluster-controller  command terminated with exit code 49: ERROR: [049]: unable to get address for 'hippo-repo-host-0.hippo-pods.postgres-operator.svc.cluster.local.': [-2] Name or service not known
user@controlnode:~/scripts/pv$ kubectl get svc -n postgres-operator
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
hippo-ha          ClusterIP   10.98.168.155   <none>        5432/TCP   9m50s
hippo-ha-config   ClusterIP   None            <none>        <none>     9m49s
hippo-pods        ClusterIP   None            <none>        <none>     9m50s
hippo-primary     ClusterIP   None            <none>        5432/TCP   9m50s
hippo-replicas    ClusterIP   10.99.199.3     <none>        5432/TCP   9m50s

[davidglassborow]

I used k3s which has the local-path-provisoner - see https://github.com/rancher/local-path-provisioner

[tony-landreth]

Hi @kuro337! I don't think you should have to create the the PVCs. But you may have to confirm that the mount directories exist and that you have permission to write to them. As davidglassborow mentioned, k3s is an easier setup to work with locally.

[andrewlecuyer]

Closing per the responses and suggestions provided above.

If you have any additional questions, or if you require any further assistance getting up-and-running, feel free to continue the conversation via the PGO project community discord server!