Cluster node restarted unexpectedly

boskiv commented 8 months ago

What happened?

timescaledb-0 timescaledb 2023-10-20 06:02:33 UTC [51474]: [653217f9.c912-3] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres application_name=pg_isready
timescaledb-0 timescaledb 2023-10-20 06:02:33 UTC [51474]: [653217f9.c912-4] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
timescaledb-0 timescaledb 2023-10-20 06:02:35,714 ERROR: Request to server https://10.128.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.128.0.1', port=443): Read timed out. (read timeout=4.999811467001564)")
timescaledb-0 timescaledb 2023-10-20 06:02:38,394 ERROR: Request to server https://10.128.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.128.0.1', port=443): Read timed out. (read timeout=2.310818386664323)")
timescaledb-0 timescaledb 2023-10-20 06:02:40,713 ERROR: Request to server https://10.128.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.128.0.1', port=443): Read timed out. (read timeout=1.919020875437127)")
timescaledb-0 timescaledb 2023-10-20 06:02:40,714 ERROR: Error communicating with DCS
timescaledb-0 timescaledb 2023-10-20 06:02:40,716 WARNING: Loop time exceeded, rescheduling immediately.
timescaledb-0 timescaledb 2023-10-20 06:02:40 UTC [93]: [65318ad1.5d-7] @,app= [00000] LOG:  received fast shutdown request
timescaledb-0 timescaledb 2023-10-20 06:02:40 UTC [93]: [65318ad1.5d-8] @,app= [00000] LOG:  aborting any active transactions

Did you expect to see something different?

How to reproduce it (as minimally and precisely as possible):

Environment

Which helm chart and what version are you using?

helm chart : 0.33.1
timescale/timescaledb-ha:pg15.4-ts2.11.2-all

What is in your values.yaml ?

values:
clusterName: "sf-cluster"
image:
  tag: pg15.4-ts2.11.2-all
persistentVolumes:
  data:
    size: 300Gi
  wal:
    size: 100Gi
pgBouncer:
  enabled: true
  config:
    ignore_startup_parameters: extra_float_digits
    server_reset_query: DISCARD ALL
    max_client_conn: 5000
    default_pool_size: 40
    pool_mode: transaction
patroni:
  bootstrap:
    dcs:
      postgresql:
        parameters:
          max_wal_size: 1GB
          min_wal_size: 512MB
          shared_preload_libraries: 'timescaledb'
          shared_buffers: 975091kB
          effective_cache_size: 2856MB
          maintenance_work_mem: 487545kB
          work_mem: 9750kB
          timescaledb.max_background_workers: 16
          max_worker_processes: 21
          max_parallel_workers_per_gather: 1
          max_parallel_workers: 2
          wal_buffers: 16MB
          default_statistics_target: 500
          random_page_cost: 1.1
          checkpoint_completion_target: 0.9
          max_connections: 500
          max_locks_per_transaction: 64
          autovacuum_max_workers: 10
          autovacuum_naptime: 10
          effective_io_concurrency: 256
          temp_file_limit: -1
prometheus:
  enabled: true
podMonitor:
  enabled: true
tolerations:
  - key: "node.k8s/role"
    operator: "Equal"
    value: "timescaledb"
    effect: "NoSchedule"

Kubernetes version information:

➜  .kube kubectl version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.5

Kubernetes cluster kind:

linode managed kubernetes 0.27 version

Anything else we need to know?:

Nothing criminal in grafana at this time

Screenshot 2023-10-20 at 09 56 58

boskiv commented 8 months ago

timescaledb-0 timescaledb 2023-10-21 05:08:41 UTC [59036]: [65335cd9.e69c-3] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres application_name=pg_isready
timescaledb-0 timescaledb 2023-10-21 05:08:41 UTC [59036]: [65335cd9.e69c-4] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [52238]: [6532ed1b.cc0e-2] @,app= [00000] LOG:  replication terminated by primary server
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [52238]: [6532ed1b.cc0e-3] @,app= [00000] DETAIL:  End of WAL reached on timeline 8 at F/450000A0.
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [52238]: [6532ed1b.cc0e-4] @,app= [08006] FATAL:  could not send end-of-streaming message to primary: SSL connection has been closed unexpectedly
timescaledb-0 timescaledb   no COPY in progress
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [46941]: [65329ac8.b75d-19] @,app= [00000] LOG:  invalid resource manager ID 109 at F/450000A0
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [59041]: [65335cdd.e6a1-1] @,app= [08006] FATAL:  could not connect to the primary server: connection to server at "10.2.3.24", port 5432 failed: server closed the connection unexpectedly
timescaledb-0 timescaledb       This probably means the server terminated abnormally
timescaledb-0 timescaledb       before or while processing the request.
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [46941]: [65329ac8.b75d-20] @,app= [00000] LOG:  waiting for WAL to become available at F/450000B8
timescaledb-0 timescaledb 2023-10-21 05:08:46 UTC [59044]: [65335cde.e6a4-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=60972
timescaledb-0 timescaledb 2023-10-21 05:08:46 UTC [59044]: [65335cde.e6a4-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:08:46 UTC [59044]: [65335cde.e6a4-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:08:46 UTC [59044]: [65335cde.e6a4-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=60972
timescaledb-0 timescaledb 2023-10-21 05:08:50 UTC [59047]: [65335ce2.e6a7-1] @,app= [00000] LOG:  started streaming WAL from primary at F/45000000 on timeline 8
timescaledb-0 timescaledb 2023-10-21 05:08:56 UTC [59054]: [65335ce8.e6ae-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=32880
timescaledb-0 timescaledb 2023-10-21 05:08:56 UTC [59054]: [65335ce8.e6ae-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:08:56 UTC [59054]: [65335ce8.e6ae-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:08:56 UTC [59054]: [65335ce8.e6ae-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=32880
timescaledb-0 timescaledb 2023-10-21 05:08:56,481 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:56,483 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:08:56,586 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:56,588 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:08:57,242 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:57,245 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:08:58,199 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:58,202 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:08:59,787 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:59,789 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:01,504 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:01,507 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:02,661 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:02,666 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:04,140 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:04,143 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:04,143 ERROR: Error communicating with DCS
timescaledb-0 timescaledb 2023-10-21 05:09:05 UTC [59061]: [65335cf1.e6b5-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=56338
timescaledb-0 timescaledb 2023-10-21 05:09:05 UTC [59061]: [65335cf1.e6b5-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:05 UTC [59061]: [65335cf1.e6b5-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:05 UTC [59061]: [65335cf1.e6b5-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.007 user=standby database= host=127.0.0.1 port=56338
timescaledb-0 timescaledb 2023-10-21 05:09:05,955 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:05,958 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:06,801 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:06,804 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:07,327 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:07,330 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:08,324 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:08,327 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:09,450 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:09,453 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:10,557 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:10,560 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:11 UTC [59068]: [65335cf7.e6bc-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
timescaledb-0 timescaledb 2023-10-21 05:09:11 UTC [59068]: [65335cf7.e6bc-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authenticated: identity="postgres" method=peer (/var/lib/postgresql/data/pg_hba.conf:3)
timescaledb-0 timescaledb 2023-10-21 05:09:11 UTC [59068]: [65335cf7.e6bc-3] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres application_name=pg_isready
timescaledb-0 timescaledb 2023-10-21 05:09:11 UTC [59068]: [65335cf7.e6bc-4] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
timescaledb-0 timescaledb 2023-10-21 05:09:11,885 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:11,887 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:12,982 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:12,984 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:14,699 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:14,700 ERROR: Error communicating with DCS
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59070]: [65335cfb.e6be-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=57358
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59070]: [65335cfb.e6be-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59070]: [65335cfb.e6be-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59070]: [65335cfb.e6be-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=57358
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59077]: [65335cfb.e6c5-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=57360
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59077]: [65335cfb.e6c5-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59077]: [65335cfb.e6c5-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59077]: [65335cfb.e6c5-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=57360
timescaledb-0 timescaledb 2023-10-21 05:09:15,959 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:15,962 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:16,175 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:16,177 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:17,111 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:17,114 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:18,060 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:18,064 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:18,868 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:18,870 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:20,094 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:20,097 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:21,412 ERROR: ObjectCache.run ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
timescaledb-0 timescaledb 2023-10-21 05:09:21,413 ERROR: ObjectCache.run ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
timescaledb-0 timescaledb server signaled
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [46937]: [65329ac8.b759-8] @,app= [00000] LOG:  received SIGHUP, reloading configuration files
timescaledb-0 timescaledb Sourcing /home/postgres/.pod_environment
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [46937]: [65329ac8.b759-9] @,app= [00000] LOG:  parameter "primary_conninfo" changed to "user=standby passfile=/home/postgres/.pgpass.patroni host=10.2.5.23 port=5432 sslmode=prefer application_name=timescaledb-0 gssencmode=prefer channel_binding=prefer"
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [46941]: [65329ac8.b75d-21] @,app= [00000] LOG:  WAL receiver process shutdown requested
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59047]: [65335ce2.e6a7-2] @,app= [57P01] FATAL:  terminating walreceiver process due to administrator command
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59146]: [65335d01.e70a-1] @,app= [00000] LOG:  fetching timeline history file for timeline 9 from primary server
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59147]: [65335d01.e70b-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=45530
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59146]: [65335d01.e70a-2] @,app= [08P01] FATAL:  could not start WAL streaming: ERROR:  replication slot "timescaledb_0" does not exist
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59147]: [65335d01.e70b-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59147]: [65335d01.e70b-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59147]: [65335d01.e70b-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.007 user=standby database= host=127.0.0.1 port=45530
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [46941]: [65329ac8.b75d-22] @,app= [00000] LOG:  new target timeline is 9
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59158]: [65335d01.e716-1] @,app= [08P01] FATAL:  could not start WAL streaming: ERROR:  replication slot "timescaledb_0" does not exist
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59163]: [65335d01.e71b-1] @,app= [08P01] FATAL:  could not start WAL streaming: ERROR:  replication slot "timescaledb_0" does not exist
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [46941]: [65329ac8.b75d-23] @,app= [00000] LOG:  waiting for WAL to become available at F/450000B8
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59166]: [65335d02.e71e-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=45534
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59166]: [65335d02.e71e-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59166]: [65335d02.e71e-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59166]: [65335d02.e71e-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=45534
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59167]: [65335d02.e71f-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=45538
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59167]: [65335d02.e71f-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59167]: [65335d02.e71f-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59167]: [65335d02.e71f-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.005 user=standby database= host=127.0.0.1 port=45538
timescaledb-0 timescaledb 2023-10-21 05:09:24 UTC [59168]: [65335d04.e720-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=45550

boskiv commented 8 months ago

no pods restarted actually, but postgres inside

timescale / helm-charts

Cluster node restarted unexpectedly #626