search

cloudnative-pg / cloudnative-pg

CloudNativePG is a comprehensive platform designed to seamlessly manage PostgreSQL databases within Kubernetes environments, covering the entire operational lifecycle from initial deployment to ongoing maintenance
https://cloudnative-pg.io
Apache License 2.0
4.45k stars 302 forks source link

[Bug]: Node restarts may render cluster broken #4393

Closed christiaangoossens closed 5 months ago

christiaangoossens commented 5 months ago

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Contact Details

No response

Version

1.23.0

What version of Kubernetes are you using?

v1.28.5+k3s1

What is your Kubernetes environment?

Self-managed: kind (evaluation)

How did you install the operator?

Helm

What happened?

Related to https://github.com/cloudnative-pg/cloudnative-pg/issues/3091 and https://github.com/cloudnative-pg/cloudnative-pg/issues/3810.

My cluster was in a state where out of three instances, only one was running (instance 6), while 7 was unready and 8 was creating. It did not bring itself up, so after taking a backup, I ran the script from https://github.com/cloudnative-pg/cloudnative-pg/issues/3810#issuecomment-1945759481, to see if the primary would be recreated.

Current state:

$ kubectl get pvc
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-db-6   Bound    pvc-d8b83106-827c-45c1-9f7b-728b2912f0d5   10Gi       RWO            proxmox-data   44d

$ kubectl get pods
test-db-8-join-w4glp         0/1     Pending   0             3m36s

$ kubectl describe 
Name:         test-db
Namespace:    app-test
Labels:       app.kubernetes.io/instance=customer-app-test
Annotations:  <none>
API Version:  postgresql.cnpg.io/v1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2024-01-06T17:20:56Z
  Generation:          6
  Resource Version:    79783390
  UID:                 efdaed68-51d6-415e-a714-825607904fd5
Spec:
  Affinity:
    Pod Anti Affinity Type:  preferred
  Bootstrap:
    Initdb:
      Database:             app
      Encoding:             UTF8
      Locale C Type:        C
      Locale Collate:       C
      Owner:                app
  Enable PDB:               true
  Enable Superuser Access:  false
  Failover Delay:           0
  Image Name:               ghcr.io/cloudnative-pg/postgresql:16.1
  Instances:                3
  Log Level:                info
  Max Sync Replicas:        0
  Min Sync Replicas:        0
  Monitoring:
    Custom Queries Config Map:
      Key:                    queries
      Name:                   cnpg-default-monitoring
    Disable Default Queries:  false
    Enable Pod Monitor:       false
  Postgres GID:               26
  Postgres UID:               26
  Postgresql:
    Parameters:
      archive_mode:                on
      archive_timeout:             5min
      dynamic_shared_memory_type:  posix
      log_destination:             csvlog
      log_directory:               /controller/log
      log_filename:                postgres
      log_rotation_age:            0
      log_rotation_size:           0
      log_truncate_on_rotation:    false
      logging_collector:           on
      max_parallel_workers:        32
      max_replication_slots:       32
      max_worker_processes:        32
      shared_memory_type:          mmap
      shared_preload_libraries:
      ssl_max_protocol_version:    TLSv1.3
      ssl_min_protocol_version:    TLSv1.3
      wal_keep_size:               512MB
      wal_level:                   logical
      wal_log_hints:               on
      wal_receiver_timeout:        5s
      wal_sender_timeout:          5s
    Sync Replica Election Constraint:
      Enabled:              false
  Primary Update Method:    restart
  Primary Update Strategy:  unsupervised
  Replication Slots:
    High Availability:
      Enabled:      true
      Slot Prefix:  _cnpg_
    Synchronize Replicas:
      Enabled:        true
    Update Interval:  30
  Resources:
  Smart Shutdown Timeout:  180
  Start Delay:             3600
  Stop Delay:              1800
  Storage:
    Resize In Use Volumes:  true
    Size:                   10Gi
  Switchover Delay:         3600
Status:
  Available Architectures:
    Go Arch:  amd64
    Hash:     5156df5aa3031085ef8a0df721bdb24df2f1f79e280520d7e906cf2c6e44da7a
    Go Arch:  arm64
    Hash:     3217c3b96eaf3237a148315014e4341ee8110d849b0e773c9683b2398c3ac252
  Certificates:
    Client CA Secret:  test-db-ca
    Expirations:
      App - Test - Db - Ca:           2024-06-27 17:46:47 +0000 UTC
      App - Test - Db - Replication:  2024-06-27 17:46:47 +0000 UTC
      App - Test - Db - Server:       2024-06-27 17:46:47 +0000 UTC
    Replication TLS Secret:           test-db-replication
    Server Alt DNS Names:
      test-db-rw
      test-db-rw.app-test
      test-db-rw.app-test.svc
      test-db-r
      test-db-r.app-test
      test-db-r.app-test.svc
      test-db-ro
      test-db-ro.app-test
      test-db-ro.app-test.svc
    Server CA Secret:             test-db-ca
    Server TLS Secret:            test-db-server
  Cloud Native PG Commit Hash:    f6292625
  Cloud Native PG Operator Hash:  5156df5aa3031085ef8a0df721bdb24df2f1f79e280520d7e906cf2c6e44da7a
  Conditions:
    Last Transition Time:  2024-03-30T11:59:51Z
    Message:               Cluster Is Not Ready
    Reason:                ClusterIsNotReady
    Status:                False
    Type:                  Ready
    Last Transition Time:  2024-01-06T17:21:25Z
    Message:               Continuous archiving is working
    Reason:                ContinuousArchivingSuccess
    Status:                True
    Type:                  ContinuousArchiving
  Config Map Resource Version:
    Metrics:
      Cnpg - Default - Monitoring:  79783091
  Current Primary:                  test-db-6
  Current Primary Timestamp:        2024-04-26T12:55:35.767523Z
  Dangling PVC:
    test-db-6
  Image:  ghcr.io/cloudnative-pg/postgresql:16.1
  Instance Names:
    test-db-6
  Instances:              1
  Job Count:              1
  Latest Generated Node:  8
  Managed Roles Status:
  Phase:         Failing over
  Phase Reason:  Failing over from test-db-7 to test-db-6
  Pooler Integrations:
    Pg Bouncer Integration:
  Pvc Count:     1
  Read Service:  test-db-r
  Secrets Resource Version:
    Application Secret Version:  79783378
    Client Ca Secret Version:    60748924
    Replication Secret Version:  60748926
    Server Ca Secret Version:    60748924
    Server Secret Version:       60748925
  Switch Replica Cluster Status:
  Target Primary:            test-db-6
  Target Primary Timestamp:  2024-04-26T12:55:13.048567Z
  Timeline ID:               6
  Topology:
    Successfully Extracted:  true
  Write Service:             test-db-rw
Events:
  Type    Reason                       Age    From            Message
  ----    ------                       ----   ----            -------
  Normal  UpdatingPodDisruptionBudget  8m56s  cloudnative-pg  Updating PodDisruptionBudget test-db-primary
  Normal  UpdatingPodDisruptionBudget  8m56s  cloudnative-pg  Updating PodDisruptionBudget test-db
  Normal  UpdatingServiceAccount       8m56s  cloudnative-pg  Updating ServiceAccount

In this describe, it is clear that it is aware of the dangling PVC, but it does not show to have an intention to recreate it.

The operator log does not have any entries relating to this cluster.

In the end, the cluster is still in a broken state, even if it's aware of the dangling PV.

Cluster resource

No response

Relevant log output

No response

Code of Conduct

christiaangoossens commented 5 months ago

For someone who has the same problem: check if there are still jobs open, in my case the join8 job was still there, even if the pod wasn't. After deleting that, the pod 6 was recreated and everything worked again.