RossVertizan
commented
3 years ago Update
I identified four issues above:
- No data in the backup bucket
- Cluster pod restarts with every sync
- Failed to watch *v1.PostgresTeam
- Postgres is down
Since number 4 seemed to be the most serious I decided to look at that first. From the logs above the obvious cause is:
FATAL: could not write lock file "postmaster.pid": No space left on deviceSo, lets find where the disk space is full.
kubectl exec --stdin --tty vitaq-coverage-cluster-0 -- /bin/bash
>df -h
root@vitaq-coverage-cluster-0:/home/postgres# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 95G 18G 77G 19% /
tmpfs 64M 0 64M 0% /dev
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 95G 18G 77G 19% /etc/hosts
tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/sdc 976M 960M 0 100% /home/postgres/pgdata
tmpfs 7.9G 12K 7.9G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 7.9G 0 7.9G 0% /proc/acpi
tmpfs 7.9G 0 7.9G 0% /proc/scsi
tmpfs 7.9G 0 7.9G 0% /sys/firmware
>cd /home/postgres/pgdata/pgroot/data
>du -k
4 ./pg_commit_ts
4 ./pg_dynshmem
4 ./pg_stat_tmp
4 ./pg_logical/mappings
4 ./pg_logical/snapshots
16 ./pg_logical
4 ./pg_serial
12 ./pg_xact
4 ./pg_wal/archive_status
933984 ./pg_wal
12 ./pg_multixact/offsets
12 ./pg_multixact/members
28 ./pg_multixact
4 ./pg_notify
4 ./pg_twophase
9380 ./base/16839
7732 ./base/13384
8836 ./base/1
10464 ./base/13385
36416 ./base
48 ./pg_subtrans
576 ./global
4 ./pg_snapshots
4 ./pg_tblspc
16 ./pg_stat
4 ./pg_replslot
971240 .It is obvious from this and from looking in the pg_wal directory that the container disk space has filled up because wal can't write the backups to the Google Cloud Storage, so I need to get that fixed to solve this problem (I hope).
To work this through, I went to the administrator section of the Zalando Postgres operator docs and then to the section "Google Cloud Platform Setup".
The first thing is to create a GCP Service Account on the Google Cloud Platform web page. Navigate to IAM and Admin -> Service Accounts. Create the Service account by following the steps and give it permissions to Google Cloud Storage. When it is created, create keys and download the JSON file.
Now create the secret as described in the documentation, my file ended up looking like this:
apiVersion: v1
kind: Secret
metadata:
name: psql-wal-creds
namespace: default
type: Opaque
stringData:
key.json: |-
{ ... }Where I have left the { ... } I pasted the entire contents of the file downloaded from Google.
Now update the aws_or_gcp section in the postgresql-operator-default-configuration.yaml.
Mine ended up looking like this:
aws_or_gcp:
additional_secret_mount: "psql-wal-creds"
additional_secret_mount_path: "/var/secrets/google"
gcp_credentials: "/var/secrets/google/key.json"
wal_gs_bucket: "postgres-operator-backup"
# aws_region: eu-central-1
# enable_ebs_gp3_migration: false
# enable_ebs_gp3_migration_max_size: 1000
# kube_iam_role: ""
# log_s3_bucket: ""
# wal_az_storage_account: ""
# wal_s3_bucket: ""I also modified the postgres-pod-config.yaml file to add the CLONE_USE_WALG_RESTORE key which was not mentioned in the guide I followed initially. The data section of my file now looks like this:
data:
# Enable to turn backups on.
USE_WALG_BACKUP: "true"
USE_WALG_RESTORE: "true"
CLONE_USE_WALG_RESTORE: "true"
WALG_GS_PREFIX: gs://postgres-operator-backup/spilo/$(SCOPE)
CLONE_WALG_GS_PREFIX: gs://postgres-operator-backup/spilo/$(CLONE_SCOPE)
WALE_BACKUP_THRESHOLD_PERCENTAGE: "100"Now it's time to deploy all of these changes:
kubectl apply -f zalando-postgres-operator\psql-wal-creds.yaml
kubectl describe secret psql-wal-creds
kubectl apply -f zalando-postgres-operator\postgresql-operator-default-configuration.yaml
kubectl apply -f zalando-postgres-operator\postgres-pod-config.yaml
kubectl rollout restart deployment postgres-operator
kubectl rollout status -w deployment postgres-operatorand when I had done this I now see that data is being written to the GCS bucket (navigate to Google Cloud Platform -> Cloud Storage -> Browser and select the bucket)
So that's issues 1 and 4 (hopefully) dealt with.
mathe-matician
bjolivot
chrisalavoinehmm
This is a logical follow on to #274. Since that issue is closed and quite old now I thought it made sense to record my experiences/problems in a new issue - I hope this is OK. Sorry this is so long.
Initial questions
Background
We are developing a cloud-based product that needs highly available PostgreSql. I found the Postgres Operator and am blown away by what it can do for me. I am using Kubernetes on GKE, so I looked for appropriate guides to walk me through the setup process. I came across #274 which lead me to Running PostgreSQL in Google Kubernetes Engine. This is a pretty good guide and I followed it making appropriate changes as I went.
Installation process
I already had Google cloud utils installed on my machine and a working Kubernetes cluster, so I skipped these steps.
Creating a GCS bucket
Easy enough I created a GCS bucket
Namespace
Initially I wanted to put all this in a separate namespace and I added namespace definitions to various YAML files, Later (after I had problems and backed the namespace stuff out) I found the correct way to install in a separate namespace
Create the Service Account
I applied the standard
operator-service-account-rbac.yamlfile:Create the pod configmap
I modified the default config map as described in the guide, so that my data section looked like this:
Question1: In the guide it says that "The variables SCOPE and CLONE_SCOPE will be defined in the containers where appropriate below (by spilo)", but they are not mentioned again. When I finally get a running cluster pod, I see it has a SCOPE env var, but there is no CLONE_SCOPE. How do I create this env var?
I created the pod config map:
Completing the postgres-operator and CRD configuration
I changed the
postgres-operator.yamlas described in the guide to use the CRD OperatorConfiguration object as described in the guide.I also edited the
postgresql-operator-default-configuration.yamlfile.docker_image: registry.opensource.zalan.do/acid/spilo-13:2.1-p1, as the guide says that anything greater than 0.2.13 will have the corrected WalG code.enable_pod_antiaffinity: truepod_environment_configmap: "postgres-pod-config"pod_service_account_name: postgres-operatorNote that this last value is different to the one in the guide, but the service account name ispostgres-operatorCreate the postgres operator
Finally I created the postgres operator:
The cluster I am using uses the new autopilot feature that Google offers, it looks after a number if cluster management tasks, such as scaling, security etc. It has minimum cpu, memory and ephemeral storage requirements for each pod it launches, so if there are resource requests defined in the manifest then it will override those values and impose the minimum values, hence the output shown above.
The guide suggests waiting before loading the default configuration file. I noticed that the postgres-operator was not running, so I looked at the logs for the postgres operator:
I noticed there was an error:
I spent a lot of time on this, to be certain I had not messed anything up with the namespaces I backed all of the namespace modifications out and deleted the namespace. I just used the default namespace.
I also loaded the CRD definition:
This had no effect on the error message shown above, I then realised that there was a second error message which was marked as fatal rather than error:
So I then went ahead and created the CRD as shown below
Create the CRD
After doing this the postgres-operator started up and ran without Crashing.
Launch a cluster
The cluster would not launch and I saw this error in the logs:
To fix the problem I went back to
postgresql-operator-default-configuration.yamland set:After this change I could get a cluster to start.
Getting the cluster to work with my app
The next step was to read the password from the created secrets for the users I had created for the database, this was simple enough, but I couldn't connect to the database. This is because by default the database is created to only accept SSL connections, as defined in the pg_hba.conf file.
Postgres Documentation for pg_hba.conf
There are a number of issues already covering this topic #482, #868 and #330, I followed instructions from the latter to disable the SSL.
At this point I was able to connect to the database and things appeared to be working. However, I noticed some remaining issues.
Remaining issues
After all of the above, I essentially have a working database, but there are a couple of issues with it. What has gone before this point is really background, but there may be things that are worth commenting on or that I have done wrong. What follows are real issues that I would like assistance with, I am happy to file separate issues for each of them, if that works better.
1. No data in the backup bucket
Despite what the guide says, I am not seeing anything get saved in the bucket that I created for backup. Though as far as I am aware I followed all of the instructions.
The bucket is defined in the file postgres-pod-config.yaml
As mentioned previously, I see $SCOPE get defined in a cluster pod, but not $CLONE_SCOPE, how do I enable/create it?
Do I need to complete these sections in postgresql-operator-default-configuration.yaml, this is not mentioned in the guide, but would seem to be a logical requirement. Where do I get the GCP credentials from and where to I store them? I'm really lost with this.
There is also this section:
What is logical backup? Do I need to complete this? I'm sure this is simple enough, but I could use some clear instructions about what to do. Thank you.
2. Cluster pod restarts with every sync
This is a topic that has many issues relating to it. #1449 #1447 #1397 #1478 #1482 and #1485 (and maybe more). This is the output I see in the logs of the postgres-operator pod:
I believe this is happening because the GKE autopilot is changing the settings on the instance which then leads to this:
the way to get around this would be set everything to be the same as autopilot will set, so there is a match. I have implemented this in the
operatorconfiguration.crd.yamlfile and I have re-applied it (is this all I need to do to have it re-read?).The problem with this is that autopilot is setting a value for ephemeral storage, but there is no way for me to set this in the
operatorconfiguration.crd.yamlfile. Any thoughts?3. Failed to watch *v1.PostgresTeam
I see many lines in the log of the postgres-operator pod
kubectl logs postgres-operator-76895fc894-j9tlqthat say:What do I need to do to fix this. Note that I changed the team name from acid, do I need to define a team somewhere? Do I need to load the PostgresTeam CRD ?
4. Postgres is down
Since I've been writing this, I've just noticed that when I run logs on the cluster instance
kubectl logs vitaq-coverage-cluster-0I get lots of blocks of output like this:What has happened? Which device is full? Is this related to some of the earlier issues?
Any help would be great.