github-actions[bot]
commented
5 months ago Closed mcfly722 closed 5 months ago
github-actions[bot]
commented
5 months ago 
dragomirp
commented
5 months ago Hi, @mcfly722, could you try resolving the error and see if the retry works? We can handle the specific exception, but I'm wondering if there is a deeper problem here.
dragomirp
commented
5 months ago Hi, @mcfly722, latest 14/edge (rev. 393) should handle the exception. Please give it a try.
mcfly722
commented
5 months ago It looks like it still not works properly.
It looped with same errors.
Guy's is there any stable version for Ubuntu 22.04 where are no this bugs? Maybe for Ubuntu 20.04?
ubuntu@jujunode1-tst:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
postgresql-maas lxd-remote-default lxd-remote/default 3.4.2 unsupported 08:59:29Z
App Version Status Scale Charm Channel Rev Exposed Message
postgresql waiting 1 postgresql 14/edge 393 no waiting to start PostgreSQL
Unit Workload Agent Machine Public address Ports Message
postgresql/0* waiting idle 0 240.9.0.97 waiting to start PostgreSQL
Machine State Address Inst id Base AZ Message
0 started 240.9.0.97 juju-77753d-0 ubuntu@22.04 Runningubuntu@jujunode1-tst:~$ juju show-status-log postgresql/0
Time Type Status Message
10 Apr 2024 08:34:50Z workload maintenance installing PostgreSQL
10 Apr 2024 08:38:24Z juju-unit error hook failed: "install"
10 Apr 2024 08:38:29Z workload maintenance installing charm software
10 Apr 2024 08:38:30Z juju-unit executing running install hook
10 Apr 2024 08:38:34Z workload maintenance installing PostgreSQL
10 Apr 2024 08:38:36Z workload waiting waiting to start PostgreSQL
10 Apr 2024 08:38:39Z juju-unit executing running restart-relation-created hook
10 Apr 2024 08:39:48Z juju-unit error hook failed: "restart-relation-created"
10 Apr 2024 08:39:53Z juju-unit executing running restart-relation-created hook
10 Apr 2024 08:41:19Z juju-unit error hook failed: "restart-relation-created"
10 Apr 2024 08:41:24Z juju-unit executing running restart-relation-created hook
10 Apr 2024 08:42:30Z juju-unit executing running leader-elected hook
10 Apr 2024 08:44:38Z juju-unit executing running config-changed hook
10 Apr 2024 08:45:44Z juju-unit executing running start hook
10 Apr 2024 08:46:52Z juju-unit executing running database-peers-relation-changed hook
10 Apr 2024 08:48:04Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 08:48:10Z juju-unit executing running database-peers-relation-changed hook
10 Apr 2024 08:48:20Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 08:48:26Z juju-unit executing running database-peers-relation-changed hook
10 Apr 2024 09:00:23Z juju-unit idleubuntu@jujunode1-tst:~$ juju debug-log --include postgresql/0
unit-postgresql-0: 08:41:16 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:41:16 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:41:17 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:41:18 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:41:19 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:41:19 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
unit-postgresql-0: 08:41:19 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
unit-postgresql-0: 08:41:24 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
unit-postgresql-0: 08:42:29 INFO juju.worker.uniter.operation ran "restart-relation-created" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 08:42:29 INFO juju.worker.uniter found queued "leader-elected" hook
unit-postgresql-0: 08:44:38 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 08:45:44 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 08:45:44 INFO juju.worker.uniter found queued "start" hook
unit-postgresql-0: 08:45:47 INFO unit.postgresql/0.juju-log Running legacy hooks/start.
unit-postgresql-0: 08:46:51 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 08:47:59 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:48:00 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 08:48:01 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:48:01 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:48:01 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:48:01 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:48:03 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:48:03 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:48:04 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-0: 08:48:09 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-0: 08:48:13 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:48:14 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:48:15 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:48:15 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:48:17 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:48:19 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:48:19 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:48:20 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-0: 08:48:25 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-0: 08:49:30 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 08:49:58 INFO juju.worker.uniter unit "postgresql/0" shutting down: catacomb 0xc0008a7b00 is dying
unit-postgresql-0: 08:49:59 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:50:00 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:50:04 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:50:06 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:50:06 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:50:06 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:50:11 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:50:11 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:50:13 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:50:15 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 08:50:17 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 08:50:18 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:50:18 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:50:18 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:50:18 INFO juju.worker.logger logger worker stopped
unit-postgresql-0: 08:50:18 INFO juju.worker.uniter unit "postgresql/0" shutting down: failed to initialize uniter for "unit-postgresql-0": connection is shut down
unit-postgresql-0: 08:50:21 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 08:50:22 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 08:50:22 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:50:22 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:50:22 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:50:24 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:50:25 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:51:00 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:51:00 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:51:02 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:51:04 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:51:04 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:51:05 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:51:09 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:51:09 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:52:13 INFO juju.worker.uniter unit "postgresql/0" shutting down: catacomb 0xc0010e6900 is dying
unit-postgresql-0: 08:52:14 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:52:14 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:52:15 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 08:52:16 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:52:16 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:52:16 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:52:19 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:52:19 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:54:01 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:54:01 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 08:54:02 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:54:03 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:54:03 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:54:03 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:54:09 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:54:09 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:54:17 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:54:18 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 08:54:20 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:54:20 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:54:20 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:54:23 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:54:23 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 08:58:02 INFO juju.worker.uniter unit "postgresql/0" shutting down: catacomb 0xc0010e7b00 is dying
unit-postgresql-0: 08:58:02 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 08:58:02 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:58:04 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 08:58:05 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 08:58:05 INFO juju.worker.logger logger worker started
unit-postgresql-0: 08:58:05 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 08:58:08 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 08:58:08 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 09:00:15 INFO juju.worker.uniter unit "postgresql/0" shutting down: catacomb 0xc000b55b00 is dying
unit-postgresql-0: 09:00:15 INFO juju Starting unit workers for "postgresql/0"
unit-postgresql-0: 09:00:15 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.9.0.100:17070"
unit-postgresql-0: 09:00:16 INFO juju.worker.apicaller [cfbd17] "unit-postgresql-0" successfully connected to "240.7.0.98:17070"
unit-postgresql-0: 09:00:17 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 09:00:17 INFO juju.worker.logger logger worker started
unit-postgresql-0: 09:00:18 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-0: 09:00:21 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 09:00:21 INFO juju.worker.uniter hooks are retried trueHi, @mcfly722, from the status log it looks like the first hook that failed is install. Can you check if the charmed-postgresql snap is installed in the unit? Can you get the traceback of the failure in the install hook?
mcfly722
commented
5 months ago Can you get the traceback of the failure in the install hook? How to do it?
From the third try I finally deployed single node. I also unit-added second one and third to cluster, but it becomes totally unstable. Sometimes I see two Primary nodes (it never shows me secondary one), sometimes some hooks fail, somtimes it has state *agent lost, see 'juju show-status-log postgresql/'**
It's absolutelly unstable :(
My current model:
ubuntu@jujunode1-tst:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
postgresql-maas lxd-remote-default lxd-remote/default 3.4.2 unsupported 13:48:31Z
App Version Status Scale Charm Channel Rev Exposed Message
data-integrator active 2/3 data-integrator latest/stable 19 no
pgbouncer 1.21.0 blocked 2/3 pgbouncer 1/edge 142 no PgBouncer service pgbouncer-pgbouncer@0 not running
postgresql 14.11 waiting 2/3 postgresql 14/edge 393 no awaiting for member to start
Unit Workload Agent Machine Public address Ports Message
data-integrator/0 unknown lost 0 240.9.0.179 agent lost, see 'juju show-status-log data-integrator/0'
pgbouncer/2 unknown lost 240.9.0.179 agent lost, see 'juju show-status-log pgbouncer/2'
data-integrator/1 active idle 1 240.81.0.221
pgbouncer/1 blocked idle 240.81.0.221 6432/tcp PgBouncer service pgbouncer-pgbouncer@0 not running
data-integrator/2* active idle 2 240.7.0.109
pgbouncer/0* blocked idle 240.7.0.109 6432/tcp PgBouncer service pgbouncer-pgbouncer@0 not running
postgresql/0 unknown lost 0 240.9.0.179 5432/tcp agent lost, see 'juju show-status-log postgresql/0'
postgresql/1 active idle 1 240.81.0.221 5432/tcp
postgresql/2* active idle 2 240.7.0.109 5432/tcp
Machine State Address Inst id Base AZ Message
0 started 240.9.0.179 juju-a1b185-0 ubuntu@22.04 Running
1 started 240.81.0.221 juju-a1b185-1 ubuntu@22.04 Running
2 started 240.7.0.109 juju-a1b185-2 ubuntu@22.04 RunningI see in log that internal scripts tries to do something, and automatically restarts elections, but no luck.
juju debug-log --include postgresql/2
unit-postgresql-2: 12:29:26 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:29:26 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:29:41 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:29:41 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:29:46 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 12:29:52 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 12:29:58 INFO juju.worker.uniter.operation ran "upgrade-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:30:05 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:30:13 INFO juju.worker.uniter.operation ran "upgrade-relation-joined" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:30:20 INFO juju.worker.uniter.operation ran "upgrade-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:29:23 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:29:23 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:29:26 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:34:13 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:37:18 ERROR unit.postgresql/2.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-postgresql-2/charm/./src/charm.py", line 1589, in <module>
main(PostgresqlOperatorCharm)
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/ops/main.py", line 544, in main
manager.run()
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/ops/main.py", line 520, in run
self._emit()
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/ops/main.py", line 509, in _emit
_emit_charm_event(self.charm, self.dispatcher.event_name)
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/ops/main.py", line 143, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/ops/framework.py", line 352, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/ops/framework.py", line 851, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/ops/framework.py", line 941, in _reemit
custom_handler(event)
File "/var/lib/juju/agents/unit-postgresql-2/charm/./src/charm.py", line 1225, in _on_update_status
if self._handle_workload_failures():
File "/var/lib/juju/agents/unit-postgresql-2/charm/./src/charm.py", line 1283, in _handle_workload_failures
if not self._patroni.member_started and self._patroni.is_member_isolated:
File "/var/lib/juju/agents/unit-postgresql-2/charm/src/cluster.py", line 424, in is_member_isolated
return len(cluster_status.json()["members"]) == 0
File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
unit-postgresql-2: 12:37:18 ERROR juju.worker.uniter.operation hook "update-status" (via hook dispatching script: dispatch) failed: exit status 1
unit-postgresql-2: 12:37:18 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 12:37:28 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 12:39:34 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:39:34 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:39:34 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:39:34 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:39:35 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:39:35 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 12:39:35 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 12:39:35 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:39:35 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:39:35 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:39:35 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:39:37 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:39:37 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:39:38 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 12:39:43 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 12:40:51 INFO unit.postgresql/2.juju-log restarted PostgreSQL because it was not running
unit-postgresql-2: 12:40:51 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:40:58 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:41:13 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:42:23 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:42:23 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:42:24 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:42:24 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:42:24 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": cannot create relation state tracker: connection is shut down
unit-postgresql-2: 12:42:24 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 12:42:24 ERROR juju.worker.dependency "hook-retry-strategy" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 12:42:24 ERROR juju.worker.dependency "migration-inactive-flag" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 12:42:24 ERROR juju.worker.dependency "migration-minion" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 12:42:24 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 12:42:24 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:42:24 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:42:24 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:42:24 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:42:25 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:42:25 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:42:26 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 12:42:32 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 12:42:57 WARNING unit.postgresql/2.database-peers-relation-changed error: snap "charmed-postgresql" has no "exporter" configuration option
unit-postgresql-2: 12:42:57 WARNING unit.postgresql/2.juju-log database-peers:0: Early exit update_config: Trying to reset metrics service with no configuration set
unit-postgresql-2: 12:43:20 WARNING unit.postgresql/2.database-peers-relation-changed error: snap "charmed-postgresql" has no "exporter" configuration option
unit-postgresql-2: 12:43:20 WARNING unit.postgresql/2.juju-log database-peers:0: Early exit update_config: Trying to reset metrics service with no configuration set
unit-postgresql-2: 12:43:54 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:43:54 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:43:59 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:44:00 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:44:00 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:44:02 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:44:02 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 12:44:02 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 12:44:02 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:44:04 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:44:04 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:44:08 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:44:11 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:44:12 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:44:13 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 12:44:19 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 12:44:22 INFO unit.postgresql/2.juju-log database-peers:0: Starting cluster topology observer process
unit-postgresql-2: 12:44:23 INFO unit.postgresql/2.juju-log database-peers:0: Started cluster topology observer process with PID 14392
unit-postgresql-2: 12:45:37 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:45:42 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:46:22 INFO juju.worker.uniter unit "postgresql/2" shutting down: catacomb 0xc00060ed80 is dying
unit-postgresql-2: 12:46:23 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:46:24 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:46:26 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:46:27 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:46:27 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:46:30 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:46:42 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:46:42 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:46:51 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:47:57 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:47:58 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:48:01 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:48:01 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:48:02 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 12:48:02 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 12:48:03 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:48:05 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:48:05 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:48:06 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:48:13 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:48:14 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:48:21 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:48:32 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:48:32 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:48:38 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:48:39 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:48:39 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:48:40 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 12:48:40 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 12:48:42 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:48:43 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:48:43 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:48:43 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:48:46 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:48:46 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:48:53 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:49:02 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:49:02 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:49:02 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:49:02 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:49:03 ERROR juju.worker.logger connection is shut down
unit-postgresql-2: 12:49:03 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 12:49:03 ERROR juju.worker.dependency "migration-minion" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 12:49:03 ERROR juju.worker.dependency "api-address-updater" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 12:49:03 ERROR juju.worker.dependency "migration-inactive-flag" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 12:49:03 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 12:49:03 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 12:49:03 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:49:08 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:49:08 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:49:09 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:49:14 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:49:14 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:49:21 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:53:25 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:53:26 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:53:32 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:53:32 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:53:38 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:53:52 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:53:53 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:54:02 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 12:57:01 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 12:57:01 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:57:03 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:57:07 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:57:07 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:57:09 ERROR juju.worker.dependency "meter-status" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 12:57:09 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 12:57:09 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": cannot create relation state tracker: connection is shut down
unit-postgresql-2: 12:57:09 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 12:57:09 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 12:57:11 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 12:57:12 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 12:57:13 INFO juju.worker.logger logger worker started
unit-postgresql-2: 12:57:16 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 12:57:23 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 12:57:24 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 12:57:33 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:03:16 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:08:34 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:08:58 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:09:08 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:09:21 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:09:32 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:09:43 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:10:24 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:10:54 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:11:19 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:11:31 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:12:32 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:12:33 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:12:40 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:12:41 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:12:41 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:12:47 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:12:54 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:12:54 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:12:57 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:13:02 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:13:11 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 13:13:11 ERROR juju.worker.dependency "leadership-tracker" manifold worker returned unexpected error: error while postgresql/2 waiting for postgresql leadership release: error blocking on leadership release: connection is shut down
unit-postgresql-2: 13:13:13 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:13:13 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:13:17 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:13:22 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:13:22 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:13:25 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:13:30 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:13:31 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:13:34 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:13:34 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:13:35 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:13:53 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:13:54 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:13:56 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:14:01 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:14:15 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:14:15 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:14:20 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:14:20 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:14:20 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:14:34 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:14:35 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:14:36 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:14:36 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:14:36 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": cannot create relation state tracker: connection is shut down
unit-postgresql-2: 13:14:36 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 13:14:36 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:14:39 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:14:46 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:14:47 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:14:51 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:14:56 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:14:56 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:14:59 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:15:05 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:15:15 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:15:15 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:15:25 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:15:39 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:15:39 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:15:41 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:15:50 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:15:51 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:15:59 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:16:05 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:16:20 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:16:29 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:16:41 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:16:41 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:16:45 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:16:50 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:16:50 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:16:51 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:17:02 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:17:02 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:17:12 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:17:24 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:17:25 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:17:30 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:17:31 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:17:32 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:17:32 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 13:17:32 ERROR juju.worker.dependency "leadership-tracker" manifold worker returned unexpected error: leadership failure: error making a leadership claim: connection is shut down
unit-postgresql-2: 13:17:32 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 13:17:32 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 13:17:36 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:17:37 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:17:37 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:17:39 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:17:48 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:17:48 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:17:58 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:18:44 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:18:48 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:18:49 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:18:50 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:18:50 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:18:51 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:18:55 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:18:56 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:19:08 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:19:28 INFO juju.worker.uniter found queued "leader-elected" hook
unit-postgresql-2: 13:20:08 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:24:08 INFO unit.postgresql/2.juju-log database-peers:0: Reconfiguring cluster
unit-postgresql-2: 13:25:13 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:25:18 INFO unit.postgresql/2.juju-log database-peers:0: Reconfiguring cluster
unit-postgresql-2: 13:26:20 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:28:29 INFO unit.postgresql/2.juju-log database-peers:0: Reconfiguring cluster
unit-postgresql-2: 13:29:30 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:30:29 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:30:30 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:30:36 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:30:41 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:30:41 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:30:45 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 13:30:45 ERROR juju.worker.dependency "migration-minion" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 13:30:45 ERROR juju.worker.dependency "meter-status" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 13:30:45 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 13:30:45 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 13:30:51 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:30:54 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:30:54 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:30:58 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:31:07 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:31:08 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:31:10 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:31:15 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-postgresql-2: 13:31:23 ERROR unit.postgresql/2.juju-log Failed to list PostgreSQL database users: could not translate host name "None" to address: Temporary failure in name resolution
unit-postgresql-2: 13:31:26 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:31:37 ERROR juju.worker.dependency "migration-minion" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 13:31:37 ERROR juju.worker.dependency "hook-retry-strategy" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 13:31:37 ERROR juju.worker.dependency "secret-drain-worker" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 13:31:37 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 13:31:37 ERROR juju.worker.dependency "logging-config-updater" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 13:31:37 ERROR juju.worker.dependency "migration-inactive-flag" manifold worker returned unexpected error: connection is shut down
unit-postgresql-2: 13:31:42 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:31:44 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:31:50 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:31:50 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:31:50 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:32:02 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:32:03 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:32:06 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:32:11 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:32:16 INFO unit.postgresql/2.juju-log database-peers:0: Reconfiguring cluster
unit-postgresql-2: 13:32:17 INFO unit.postgresql/2.juju-log database-peers:0: not all members are ready
unit-postgresql-2: 13:32:17 INFO unit.postgresql/2.juju-log database-peers:0: Deferring reconfigure: another member doing sync right now
unit-postgresql-2: 13:32:24 WARNING unit.postgresql/2.juju-log database-peers:0: Failed to connect to PostgreSQL.
unit-postgresql-2: 13:32:28 WARNING unit.postgresql/2.juju-log database-peers:0: Failed to connect to PostgreSQL.
unit-postgresql-2: 13:32:32 WARNING unit.postgresql/2.juju-log database-peers:0: Failed to connect to PostgreSQL.
unit-postgresql-2: 13:32:35 WARNING unit.postgresql/2.juju-log database-peers:0: Failed to connect to PostgreSQL.
unit-postgresql-2: 13:32:40 WARNING unit.postgresql/2.juju-log database-peers:0: Failed to connect to PostgreSQL.
unit-postgresql-2: 13:32:46 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:33:54 INFO unit.postgresql/2.juju-log database-peers:0: Reconfiguring cluster
unit-postgresql-2: 13:33:55 INFO unit.postgresql/2.juju-log database-peers:0: not all members are ready
unit-postgresql-2: 13:33:55 INFO unit.postgresql/2.juju-log database-peers:0: Deferring reconfigure: another member doing sync right now
unit-postgresql-2: 13:34:03 WARNING unit.postgresql/2.juju-log database-peers:0: Failed to connect to PostgreSQL.
unit-postgresql-2: 13:34:27 INFO unit.postgresql/2.juju-log database-peers:0: Reconfiguring cluster
unit-postgresql-2: 13:35:49 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:35:49 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:35:56 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:35:56 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:35:57 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:36:19 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:36:20 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:36:24 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:36:25 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:36:30 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:36:47 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:36:47 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:36:56 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:36:57 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:36:57 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:37:05 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:37:08 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:37:14 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:37:14 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:37:14 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:37:23 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:37:23 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:37:26 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:37:32 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:37:51 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:37:52 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:37:55 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:37:55 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:37:57 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 13:37:57 INFO juju.worker.uniter unit "postgresql/2" shutting down: failed to initialize uniter for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 13:37:59 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:38:03 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:38:05 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:38:05 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:38:08 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:38:24 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:38:25 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:38:27 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:38:33 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:39:46 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 13:40:47 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:40:48 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:40:52 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:40:52 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:40:56 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:40:57 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:40:59 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:41:01 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:41:04 ERROR juju.worker.dependency "upgrader" manifold worker returned unexpected error: cannot set agent version for "unit-postgresql-2": connection is shut down
unit-postgresql-2: 13:41:04 INFO juju.worker.logger logger worker stopped
unit-postgresql-2: 13:41:04 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.9.0.100:17070"
unit-postgresql-2: 13:41:06 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:41:06 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:41:06 INFO juju.worker.upgrader no waiter, upgrader is done
unit-postgresql-2: 13:41:31 INFO juju.worker.uniter unit "postgresql/2" started
unit-postgresql-2: 13:41:31 INFO juju.worker.uniter hooks are retried true
unit-postgresql-2: 13:41:38 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:41:43 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-postgresql-2: 13:41:48 INFO juju Starting unit workers for "postgresql/2"
unit-postgresql-2: 13:41:49 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:41:54 INFO juju.worker.apicaller [768c83] "unit-postgresql-2" successfully connected to "240.7.0.98:17070"
unit-postgresql-2: 13:41:57 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-2: 13:41:57 INFO juju.worker.logger logger worker started
unit-postgresql-2: 13:42:01 INFO juju.worker.upgrader no waiter, upgrader is doneubuntu@jujunode1-tst:~$ juju show-status-log postgresql/0
Time Type Status Message
10 Apr 2024 13:32:05Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:32:11Z juju-unit idle
10 Apr 2024 13:33:50Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:34:28Z workload active
10 Apr 2024 13:35:15Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:35:27Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:37:26Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:37:31Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:38:22Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:38:33Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:41:38Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:41:45Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:42:19Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:42:30Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:44:53Z workload waiting awaiting for member to start
10 Apr 2024 13:45:08Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:47:37Z juju-unit error hook failed: "leader-settings-changed"
10 Apr 2024 13:47:44Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:49:02Z juju-unit error hook failed: "leader-settings-changed"
10 Apr 2024 13:49:17Z juju-unit executing running leader-settings-changed hookubuntu@jujunode1-tst:~$ juju show-status-log postgresql/1
Time Type Status Message
10 Apr 2024 13:32:38Z juju-unit error hook failed: "start"
10 Apr 2024 13:32:43Z juju-unit executing running start hook
10 Apr 2024 13:33:50Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:33:54Z juju-unit executing running database-peers-relation-changed hook for postgresql/0
10 Apr 2024 13:34:02Z workload waiting awaiting for member to start
10 Apr 2024 13:34:04Z juju-unit idle
10 Apr 2024 13:35:20Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:42:19Z juju-unit error hook failed: "leader-settings-changed"
10 Apr 2024 13:42:39Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:43:10Z juju-unit executing running database-relation-changed hook for pgbouncer/0
10 Apr 2024 13:43:21Z juju-unit executing running database-relation-changed hook for pgbouncer/1
10 Apr 2024 13:43:25Z juju-unit idle
10 Apr 2024 13:44:56Z juju-unit executing running database-peers-relation-changed hook for postgresql/0
10 Apr 2024 13:45:19Z workload active
10 Apr 2024 13:45:22Z juju-unit executing running database-relation-changed hook for pgbouncer/2
10 Apr 2024 13:45:24Z juju-unit idle
10 Apr 2024 13:47:37Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:47:40Z juju-unit idle
10 Apr 2024 13:48:41Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:48:44Z juju-unit idleubuntu@jujunode1-tst:~$ juju show-status-log postgresql/2
Time Type Status Message
10 Apr 2024 13:33:50Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:33:54Z workload maintenance reconfiguring cluster
10 Apr 2024 13:34:26Z workload active
10 Apr 2024 13:34:27Z workload maintenance reconfiguring cluster
10 Apr 2024 13:37:26Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:37:40Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:38:28Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:38:33Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:41:38Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:41:44Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:42:18Z juju-unit error hook failed: "database-peers-relation-changed"
10 Apr 2024 13:42:25Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:43:11Z juju-unit executing running database-relation-changed hook for pgbouncer/0
10 Apr 2024 13:43:25Z juju-unit executing running database-relation-changed hook for pgbouncer/1
10 Apr 2024 13:43:32Z juju-unit executing running database-peers-relation-changed hook for postgresql/1
10 Apr 2024 13:44:19Z juju-unit idle
10 Apr 2024 13:44:56Z juju-unit executing running database-peers-relation-changed hook for postgresql/0
10 Apr 2024 13:45:35Z workload active
10 Apr 2024 13:45:38Z juju-unit executing running database-relation-changed hook for pgbouncer/2
10 Apr 2024 13:48:40Z juju-unit idleubuntu@jujunode1-tst:~$ juju show-status-log pgbouncer/0
Time Type Status Message
10 Apr 2024 13:27:47Z workload maintenance Updating database relation connection information
10 Apr 2024 13:27:49Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:32:06Z juju-unit idle
10 Apr 2024 13:36:29Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:37:27Z juju-unit executing running leader-elected hook
10 Apr 2024 13:37:30Z workload maintenance Updating database relation connection information
10 Apr 2024 13:37:32Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:37:33Z workload blocked PgBouncer service pgbouncer-pgbouncer@2 not running
10 Apr 2024 13:38:25Z juju-unit idle
10 Apr 2024 13:39:48Z juju-unit executing running backend-database-relation-changed hook
10 Apr 2024 13:39:54Z workload maintenance updating PgBouncer config
10 Apr 2024 13:39:55Z workload blocked PgBouncer service pgbouncer-pgbouncer@2 not running
10 Apr 2024 13:39:55Z workload maintenance Reloading Pgbouncer
10 Apr 2024 13:39:56Z workload blocked PgBouncer service pgbouncer-pgbouncer@2 not running
10 Apr 2024 13:39:58Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:39:59Z workload maintenance Updating database relation connection information
10 Apr 2024 13:40:01Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:40:02Z workload blocked PgBouncer service pgbouncer-pgbouncer@1 not running
10 Apr 2024 13:40:02Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:48:38Z juju-unit idleubuntu@jujunode1-tst:~$ juju show-status-log pgbouncer/1
Time Type Status Message
10 Apr 2024 13:19:03Z juju-unit idle
10 Apr 2024 13:27:40Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:31:11Z juju-unit executing running start hook
10 Apr 2024 13:31:15Z workload maintenance updating PgBouncer users
10 Apr 2024 13:31:16Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:31:17Z workload blocked backend database relation not ready
10 Apr 2024 13:32:40Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:32:42Z juju-unit idle
10 Apr 2024 13:42:20Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:42:29Z juju-unit executing running backend-database-relation-changed hook
10 Apr 2024 13:42:34Z workload maintenance updating PgBouncer config
10 Apr 2024 13:42:34Z workload blocked backend database relation not ready
10 Apr 2024 13:42:34Z workload maintenance Reloading Pgbouncer
10 Apr 2024 13:42:35Z workload blocked backend database relation not ready
10 Apr 2024 13:42:37Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:42:38Z juju-unit idle
10 Apr 2024 13:47:34Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:47:37Z juju-unit idle
10 Apr 2024 13:48:44Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:48:47Z juju-unit idleubuntu@jujunode1-tst:~$ juju show-status-log pgbouncer/2
Time Type Status Message
10 Apr 2024 13:32:14Z juju-unit idle
10 Apr 2024 13:35:20Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:35:26Z juju-unit idle
10 Apr 2024 13:36:25Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:36:31Z juju-unit idle
10 Apr 2024 13:37:27Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:37:31Z juju-unit idle
10 Apr 2024 13:38:28Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:38:33Z juju-unit idle
10 Apr 2024 13:42:21Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:44:56Z juju-unit executing running backend-database-relation-changed hook
10 Apr 2024 13:45:02Z workload maintenance updating PgBouncer config
10 Apr 2024 13:45:02Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:45:02Z workload maintenance Reloading Pgbouncer
10 Apr 2024 13:45:06Z workload blocked PgBouncer service pgbouncer-pgbouncer@0 not running
10 Apr 2024 13:45:07Z juju-unit idle
10 Apr 2024 13:47:39Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:47:44Z juju-unit idle
10 Apr 2024 13:49:07Z juju-unit executing running leader-settings-changed hook
10 Apr 2024 13:49:17Z juju-unit idleHi, pgbouncer uses the same snap. Can you ssh into a failing unit and run snap list? For postgresql charm revision 393, you should have a charmed-postgresql snap with revision 104 installed. For pgbouncer charm revision 142 you should have a charmed-postgresql snap with revision 111 installed.
If the snap is not installed, try to install it manually and see if there are any errors. If the snap is already installed, check if there is a snap.charmed-postgresql.patroni service running.
The data-integrator charm doesn't install any snaps, so that it is also having issues makes me think there is some kind of general connectivity issue
mcfly722
commented
5 months ago As I understand, (correct me, please, if not) charmed-postgresql snap revision should match for postgresql-operator and pgbouncer-operator on same machine. So, question - is there any table with this information (what particular charmed-postgresql snap revision uses particular operator revision)?
I need just simple PostgreSQL14 cluster with bouncer on base=Ubuntu22.04 on lxd+juju cluster, and don't figure out, is there any legitime way to do it without manual machine intrusion?
dragomirp
commented
5 months ago Under normal circumstances the charms will install the snap with the correct revision. But since the install hook has failed, which mainly installs and configures the snap, I'm wondering if the issue is that the snap was unable to install. What the charm is doing is to call the snap CLI to install, so if calling it manually doesn't work, the charm wouldn't be able to bootstrap itself either.
dragomirp
commented
5 months ago I think the issue is that pgbouncer and postgresql are installed on the same machine. Can you try deploying data-integrator on separate machines?
mcfly722
commented
5 months ago Currently I have problem with forming PgSQL cluster on edge/rev393 (I'm trying without pgbouncer and dataintegrator).
First Primary starts, but after creating second unit, it can't join and form a cluster.
on both units in /var/log/juju/unit-postgresql-0.log i see next repetitive exceptions:
2024-04-11 11:28:27 ERROR unit.postgresql/0.juju-log server.go:325 database-peers:2: Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/model.py", line 3022, in _run
result = subprocess.run(args, **kwargs) # type: ignore
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-postgresql-0/goal-state', '--format=json')' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-postgresql-0/charm/./src/charm.py", line 1589, in <module>
main(PostgresqlOperatorCharm)
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/main.py", line 544, in main
manager.run()
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/main.py", line 520, in run
self._emit()
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/main.py", line 509, in _emit
_emit_charm_event(self.charm, self.dispatcher.event_name)
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/main.py", line 143, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/framework.py", line 352, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/framework.py", line 851, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/framework.py", line 941, in _reemit
custom_handler(event)
File "/var/lib/juju/agents/unit-postgresql-0/charm/./src/charm.py", line 505, in _on_peer_relation_changed
self._patroni.start_patroni()
File "/var/lib/juju/agents/unit-postgresql-0/charm/./src/charm.py", line 694, in _patroni
self.app.planned_units(),
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/model.py", line 432, in planned_units
return self._backend.planned_units()
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/model.py", line 3339, in planned_units
app_state = self._run('goal-state', return_output=True, use_json=True)
File "/var/lib/juju/agents/unit-postgresql-0/charm/venv/ops/model.py", line 3024, in _run
raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR connection is shut downUnfortunatelly stable versions does not starts even first Primary (JSONDecodeError error occurs (updated some external python library?))
dragomirp
commented
5 months ago Hi, are you able to deploy other charms in that model? Looks like there are connectivity issues between the controller and juju units.
mcfly722
commented
5 months ago I envestigated and really found connectivity issus dealed with incorrect MTU on lxd nodes. Thank you for the tip. Cluster successfully deployed, but there are problem with PGBouncer still exists. Looks like PGBouncer should be fixed...
juju status --model postgresql-maas --color jujunode1-tst: Fri Apr 12 08:11:36 2024
Model Controller Cloud/Region Version SLA Timestamp
postgresql-maas lxd-remote-default lxd-remote/default 3.4.2 unsupported 08:11:38Z
App Version Status Scale Charm Channel Rev Exposed Message
data-integrator active 3 data-integrator latest/stable 19 no
pgbouncer 1.21.0 blocked 3 pgbouncer 1/edge 142 no PgBouncer service pgbouncer-pgbouncer@1 not running
postgresql 14.11 active 3 postgresql 14/edge 393 no Primary
Unit Workload Agent Machine Public address Ports Message
data-integrator/0* active idle 0 240.9.0.214
pgbouncer/0 blocked idle 240.9.0.214 PgBouncer service pgbouncer-pgbouncer@1 not running
data-integrator/1 active idle 1 240.81.0.7
pgbouncer/1* blocked idle 240.81.0.7 6432/tcp PgBouncer service pgbouncer-pgbouncer@0 not running
data-integrator/2 active idle 2 240.7.0.119
pgbouncer/2 blocked idle 240.7.0.119 PgBouncer service pgbouncer-pgbouncer@0 not running
postgresql/0* active idle 0 240.9.0.214 5432/tcp Primary
postgresql/1 active idle 1 240.81.0.7 5432/tcp
postgresql/2 active idle 2 240.7.0.119 5432/tcp
Machine State Address Inst id Base AZ Message
0 started 240.9.0.214 juju-1186e1-0 ubuntu@22.04 Running
1 started 240.81.0.7 juju-1186e1-1 ubuntu@22.04 Running
2 started 240.7.0.119 juju-1186e1-2 ubuntu@22.04 RunningRegistered with this issue: https://github.com/canonical/pgbouncer-operator/issues/206
dragomirp
commented
5 months ago Glad it worked out.
Steps to reproduce
Expected behavior
All SQL instances+bouncers+data integrations is UP
Actual behavior
Versions
Operating system: Ubuntu 22.04.4 LTS
Juju CLI: 3.4.1-genericlinux-amd64
Juju agent: 3.4.0
Charm revision: 14/stable
LXD: 5.0.3 (juju cluster deployed without lxd, just snap)
Log output
Juju debug log:
Additional context
juju status output: