search

timescale / helm-charts

Configuration and Documentation to run TimescaleDB in your Kubernetes cluster
Apache License 2.0
264 stars 223 forks source link

[ISSUE] Deployment error when setting log_destination to csvlog #371

Open martinmesserli opened 2 years ago

martinmesserli commented 2 years ago

Dear TimescaleDB team,

I encountered an issue when deploying timescale/timescaledb-single.

Describe the bug When I deploy a single node TimescaleDB via Helm on GKE, I want to switch to csvlog among other settings, the following error results: WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"

To Reproduce

> helm install timescale timescale/timescaledb-single \
--create-namespace -n timescale-demo \
--set patroni.bootstrap.dcs.postgresql.parameters.log_connections=off \
--set patroni.bootstrap.dcs.postgresql.parameters.log_disconnections=off \
--set patroni.bootstrap.dcs.postgresql.parameters.log_checkpoints=off \
--set patroni.bootstrap.dcs.postgresql.parameters.log_destination=csvlog \
--set patroni.bootstrap.dcs.postgresql.parameters.logging_collector=on \
--set patroni.bootstrap.dcs.postgresql.parameters.log_directory=pg_log \
--set patroni.bootstrap.dcs.postgresql.parameters.log_filename='postgresql-%Y-%m-%d_%H%M%S.log' \
--set patroni.bootstrap.dcs.postgresql.parameters.log_file_mode=511 \
--set patroni.bootstrap.dcs.postgresql.parameters.log_rotation_age=1d \
--set patroni.bootstrap.dcs.postgresql.parameters.log_rotation_size=10MB \
--set patroni.bootstrap.dcs.postgresql.parameters.log_truncate_on_rotation=off \
--set patroni.watchdog.mode=automatic  \
--set patroni.watchdog.device='/dev/watchdog' \
--set patroni.watchdog.safety_margin=5

See specifically the following settings: --set patroni.bootstrap.dcs.postgresql.parameters.log_destination=csvlog \ --set patroni.bootstrap.dcs.postgresql.parameters.logging_collector=on \

According to the documentation when switching log_destination to csvlog, the setting logging_collector must be on, and with that other settings to (see helm install flags above).

The following error occurs (excerpt from pod log timescale-timescaledb-0) and the postgres server is not running in neither of the 3 pods (replica count 3):

CREATE EXTENSION server signaled 2022-05-24 23:32:12,373 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"

See below full pod log of timescale-timescaledb-0.

Expected behavior I expect the container timescale-timescaledb-0 and the other two pods (replica count 3) to initialize correctly and postgres server to be running.

Deployment

> kubectl get all,secret,configmap,endpoints,pvc -L role -l release=timescale --namespace timescale-demo
NAME                          READY   STATUS    RESTARTS   AGE   ROLE
pod/timescale-timescaledb-0   1/1     Running   0          22m   master
pod/timescale-timescaledb-1   1/1     Running   0          21m   replica
pod/timescale-timescaledb-2   1/1     Running   0          19m   replica

NAME                        TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)          AGE   ROLE
service/timescale           LoadBalancer   10.112.4.241   <redacted>   5432:30393/TCP   22m   master
service/timescale-config    ClusterIP      None           <none>         8008/TCP         22m   
service/timescale-replica   ClusterIP      10.112.9.229   <none>         5432/TCP         22m   replica

NAME                                     READY   AGE   ROLE
statefulset.apps/timescale-timescaledb   3/3     22m   

NAME                                         DATA   AGE   ROLE
configmap/timescale-timescaledb-patroni      1      22m   
configmap/timescale-timescaledb-pgbackrest   1      22m   
configmap/timescale-timescaledb-scripts      8      22m   

NAME                          ENDPOINTS                                               AGE   ROLE
endpoints/timescale           10.108.0.150:5432                                       21m   
endpoints/timescale-config    10.108.0.150:8008,10.108.0.151:8008,10.108.0.152:8008   22m   
endpoints/timescale-replica   10.108.0.151:5432,10.108.0.152:5432                     22m   replica

NAME                                                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   ROLE
persistentvolumeclaim/storage-volume-timescale-timescaledb-0   Bound    pvc-903754e1-2c51-4ecc-af87-f8f0a25d9be0   2Gi        RWO            standard       22m   
persistentvolumeclaim/storage-volume-timescale-timescaledb-1   Bound    pvc-c0ac62e5-0d93-4b34-b765-0a6f9d4bdaee   2Gi        RWO            standard       21m   
persistentvolumeclaim/storage-volume-timescale-timescaledb-2   Bound    pvc-2002c48f-8874-4498-9589-c2f92ac0d0b0   2Gi        RWO            standard       19m   
persistentvolumeclaim/wal-volume-timescale-timescaledb-0       Bound    pvc-6cc32a56-cd89-4a15-93cf-61703a7349c0   1Gi        RWO            standard       22m   
persistentvolumeclaim/wal-volume-timescale-timescaledb-1       Bound    pvc-a042c614-4247-4eff-a6ce-317ac96e2639   1Gi        RWO            standard       21m   
persistentvolumeclaim/wal-volume-timescale-timescaledb-2       Bound    pvc-7b697e6c-7b5b-4efc-a973-312ea1130267   1Gi        RWO            standard       19m

Logs

pod log of timescale-timescaledb-0: 2022-05-24 23:32:08 - restore_or_initdb - Invoking initdb The files belonging to this database system will be owned by user "postgres". This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8". The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok fixing permissions on existing directory /var/lib/postgresql/wal/pg_wal ... ok creating subdirectories ... ok selecting dynamic shared memory implementation ... posix selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default time zone ... Etc/UTC creating configuration files ... ok running bootstrap script ... ok performing post-bootstrap initialization ... ok syncing data to disk ... ok

Success. You can now start the database server using:

pg_ctl -D /var/lib/postgresql/data -l logfile start

/var/run/postgresql:5432 - no response 2022-05-24 23:32:09 UTC [42]: [628d6af9.2a-1] @,app= [00000] LOG: redirecting log output to logging collector process 2022-05-24 23:32:09 UTC [42]: [628d6af9.2a-2] @,app= [00000] HINT: Future log output will appear in directory "pg_log". /var/run/postgresql:5432 - accepting connections /var/run/postgresql:5432 - accepting connections 2022-05-24 23:32:10 - post_init - Creating extension TimescaleDB in template1 and postgres databases You are now connected to database "template1" as user "postgres". SET WARNING:
WELCOME TO

| () | | | \ \ | | _ _ _ | | | | | | |/ / | | | | _ \ / _ \/ __|/ __/ _ | |/ \ | | | \ | | | | | | | | | /_ \ (| (_| | | _/ |/ /| |/ / || ||| || ||\||/\_,||_|/ ____/ Running version 2.6.1 For more information on TimescaleDB, please visit the following links:

  1. Getting started: https://docs.timescale.com/timescaledb/latest/getting-started
  2. API reference documentation: https://docs.timescale.com/api/latest
  3. How TimescaleDB is designed: https://docs.timescale.com/timescaledb/latest/overview/core-concepts

Note: TimescaleDB collects anonymous reports to better understand and assist our users. For more information and how to disable, please see our docs https://docs.timescale.com/timescaledb/latest/how-to-guides/configuration/telemetry.

CREATE EXTENSION You are now connected to database "postgres" as user "postgres". SET WARNING:
WELCOME TO

| () | | | \ \ | | _ _ _ | | | | | | |/ / | | | | _ \ / _ \/ __|/ __/ _ | |/ \ | | | \ | | | | | | | | | /_ \ (| (_| | | _/ |/ /| |/ / || ||| || ||\||/\_,||_|/ ____/ Running version 2.6.1 For more information on TimescaleDB, please visit the following links:

  1. Getting started: https://docs.timescale.com/timescaledb/latest/getting-started
  2. API reference documentation: https://docs.timescale.com/api/latest
  3. How TimescaleDB is designed: https://docs.timescale.com/timescaledb/latest/overview/core-concepts

Note: TimescaleDB collects anonymous reports to better understand and assist our users. For more information and how to disable, please see our docs https://docs.timescale.com/timescaledb/latest/how-to-guides/configuration/telemetry.

CREATE EXTENSION server signaled 2022-05-24 23:32:12,373 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"

pod log of timescale-timescaledb-1: 2022-05-24 23:32:56,242 ERROR: Error creating replica using method pgbackrest: /etc/timescaledb/scripts/pgbackrest_restore.sh exited with code=1 /var/run/postgresql:5432 - no response 2022-05-24 23:34:11 UTC [53]: [628d6b73.35-1] @,app= [00000] LOG: redirecting log output to logging collector process 2022-05-24 23:34:11 UTC [53]: [628d6b73.35-2] @,app= [00000] HINT: Future log output will appear in directory "pg_log". /var/run/postgresql:5432 - accepting connections /var/run/postgresql:5432 - accepting connections

pod log of timescale-timescaledb-2: 2022-05-24 23:34:55,829 ERROR: Error creating replica using method pgbackrest: /etc/timescaledb/scripts/pgbackrest_restore.sh exited with code=1 /var/run/postgresql:5432 - no response 2022-05-24 23:34:57 UTC [29]: [628d6ba0.1d-1] @,app= [00000] LOG: redirecting log output to logging collector process 2022-05-24 23:34:57 UTC [29]: [628d6ba0.1d-2] @,app= [00000] HINT: Future log output will appear in directory "pg_log". /var/run/postgresql:5432 - accepting connections /var/run/postgresql:5432 - accepting connections

Additional context