search

canonical / postgresql-operator

A Charmed Operator for running PostgreSQL on machines
https://charmhub.io/postgresql
Apache License 2.0
8 stars 19 forks source link

Charm stuck in blocked state after failed data-integrator relation has been fixed #183

Closed gustavosr98 closed 1 year ago

gustavosr98 commented 1 year ago

Steps to reproduce

juju deploy postgresql
juju deploy data-integrator postgresql-admin --to 1
juju config postgresql-admin database-name=maas extra-user-roles=admin
juju relate postgresql postgresql-admin

Results in error (#182)

Now, lets fix the relation to make it work

juju config postgresql-admin --reset extra-user-roles
# Still not working, so I had to re-create the relation
juju remove-relation postgresql-admin postgresql
juju add-relation postgresql-admin postgresql

Expected behavior

To have postgresql workload as active instead of blocked

Actual behavior

juju status
Model    Controller  Cloud/Region         Version  SLA          Timestamp
default  lxd         localhost/localhost  2.9.42   unsupported  17:14:52-05:00

App               Version  Status   Scale  Charm            Channel    Rev  Exposed  Message
postgresql        14.7     blocked      1  postgresql       14/stable  288  no       Failed to initialize database relation
postgresql-admin           active       1  data-integrator  stable      11  no       

Unit                 Workload  Agent  Machine  Public address  Ports  Message
postgresql-admin/0*  active    idle   1        10.46.90.163           
postgresql/1*        blocked   idle   1        10.46.90.163           Failed to initialize database relation

Machine  State    Address       Inst id        Series  AZ  Message
1        started  10.46.90.163  juju-f9c688-1  jammy       Running

Versions

Operating system: Jammy Juju CLI: 2.9.43 Juju agent: 2.9.42 LXD: 5.1

Charm versions:

Log output

$ cat unit-postgresql-1.log
2023-07-12 20:43:23 INFO juju unit_agent.go:289 Starting unit workers for "postgresql/1"
2023-07-12 20:43:23 INFO juju.worker.apicaller connect.go:163 [70e3fa] "unit-postgresql-1" successfully connected to "10.46.90.219:17070"
2023-07-12 20:43:23 INFO juju.worker.apicaller connect.go:260 [70e3fa] password changed for "unit-postgresql-1"
2023-07-12 20:43:23 INFO juju.worker.apicaller connect.go:163 [70e3fa] "unit-postgresql-1" successfully connected to "10.46.90.219:17070"
2023-07-12 20:43:23 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2023-07-12 20:43:23 INFO juju.worker.logger logger.go:120 logger worker started
2023-07-12 20:43:23 INFO juju.worker.upgrader upgrader.go:216 no waiter, upgrader is done
2023-07-12 20:43:23 ERROR juju.worker.meterstatus runner.go:93 error running "meter-status-changed": charm missing from disk
2023-07-12 20:43:23 INFO juju.worker.uniter uniter.go:326 unit "postgresql/1" started
2023-07-12 20:43:23 INFO juju.worker.uniter uniter.go:634 resuming charm install
2023-07-12 20:43:23 INFO juju.worker.uniter.charm bundles.go:78 downloading ch:amd64/jammy/postgresql-288 from API server
2023-07-12 20:43:31 INFO juju.worker.uniter uniter.go:344 hooks are retried true
2023-07-12 20:43:33 INFO juju.worker.uniter.operation runhook.go:159 ran "pgdata-storage-attached" hook (via hook dispatching script: dispatch)
2023-07-12 20:43:33 INFO juju.worker.uniter.storage resolver.go:127 initial storage attachments ready
2023-07-12 20:43:33 INFO unit.postgresql/1.juju-log server.go:316 Running legacy hooks/install.
2023-07-12 20:44:21 INFO juju.worker.uniter.operation runhook.go:159 ran "install" hook (via hook dispatching script: dispatch)
2023-07-12 20:44:22 INFO juju.worker.uniter.operation runhook.go:159 ran "database-peers-relation-created" hook (via hook dispatching script: dispatch)
2023-07-12 20:44:23 INFO juju.worker.uniter.operation runhook.go:159 ran "restart-relation-created" hook (via hook dispatching script: dispatch)
2023-07-12 20:44:23 INFO juju.worker.uniter resolver.go:159 found queued "leader-elected" hook
2023-07-12 20:45:24 INFO juju.worker.uniter.operation runhook.go:159 ran "leader-elected" hook (via hook dispatching script: dispatch)
2023-07-12 20:45:25 INFO juju.worker.uniter.operation runhook.go:159 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:45:25 INFO juju.worker.uniter resolver.go:159 found queued "start" hook
2023-07-12 20:45:26 INFO unit.postgresql/1.juju-log server.go:316 Running legacy hooks/start.
2023-07-12 20:45:30 INFO juju.worker.uniter.operation runhook.go:159 ran "start" hook (via hook dispatching script: dispatch)
2023-07-12 20:45:31 INFO juju.worker.uniter.operation runhook.go:159 ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:49:33 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 20:49:33 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 20:49:34 INFO unit.postgresql/1.juju-log server.go:316 database-peers:2: Starting cluster topology observer process
2023-07-12 20:49:34 INFO unit.postgresql/1.juju-log server.go:316 database-peers:2: Started cluster topology observer process with PID 15391
2023-07-12 20:49:35 INFO juju.worker.uniter.operation runhook.go:159 ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:28 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-created" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:30 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:31 ERROR unit.postgresql/1.juju-log server.go:316 database:5: Failed to create user: syntax error at or near ";"
LINE 1: ...n-5" WITH LOGIN ENCRYPTED PASSWORD '2zHH9YZF2N1hnBdx' admin;
                                                                      ^

2023-07-12 20:50:31 ERROR unit.postgresql/1.juju-log server.go:316 database:5: 
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-1/charm/lib/charms/postgresql_k8s/v0/postgresql.py", line 174, in create_user
    cursor.execute(
psycopg2.errors.SyntaxError: syntax error at or near ";"
LINE 1: ...n-5" WITH LOGIN ENCRYPTED PASSWORD '2zHH9YZF2N1hnBdx' admin;
                                                                      ^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-1/charm/src/relations/postgresql_provider.py", line 82, in _on_database_requested
    self.charm.postgresql.create_user(user, password, extra_user_roles=extra_user_roles)
  File "/var/lib/juju/agents/unit-postgresql-1/charm/lib/charms/postgresql_k8s/v0/postgresql.py", line 179, in create_user
    raise PostgreSQLCreateUserError()
charms.postgresql_k8s.v0.postgresql.PostgreSQLCreateUserError
2023-07-12 20:50:32 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:33 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:35 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:54:31 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 20:54:31 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 20:58:46 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 20:58:46 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:03:10 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:03:10 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:08:47 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:08:47 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:13:21 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:13:21 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:14:29 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 21:17:42 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:17:43 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:21:49 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:21:49 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:27:47 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:27:48 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:32:26 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:32:26 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:36:55 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:36:55 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:41:54 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:41:54 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:47:54 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:47:54 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:53:08 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:53:09 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:57:29 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 21:57:30 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:44 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-departed" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:46 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-broken" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:49 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 22:01:49 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:59 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-created" hook (via hook dispatching script: dispatch)
2023-07-12 22:02:00 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-joined" hook (via hook dispatching script: dispatch)
2023-07-12 22:02:02 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 22:02:03 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 22:02:05 INFO juju.worker.uniter.operation runhook.go:159 ran "database-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 22:06:40 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 22:06:40 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 22:12:21 WARNING unit.postgresql/1.juju-log server.go:316 No relation: certificates
2023-07-12 22:12:21 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
$ cat unit-postgresql-admin-0.log 
2023-07-12 20:49:17 INFO juju unit_agent.go:289 Starting unit workers for "postgresql-admin/0"
2023-07-12 20:49:17 INFO juju.worker.apicaller connect.go:163 [70e3fa] "unit-postgresql-admin-0" successfully connected to "10.46.90.219:17070"
2023-07-12 20:49:17 INFO juju.worker.apicaller connect.go:260 [70e3fa] password changed for "unit-postgresql-admin-0"
2023-07-12 20:49:17 INFO juju.worker.apicaller connect.go:163 [70e3fa] "unit-postgresql-admin-0" successfully connected to "10.46.90.219:17070"
2023-07-12 20:49:17 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2023-07-12 20:49:17 INFO juju.worker.logger logger.go:120 logger worker started
2023-07-12 20:49:17 INFO juju.worker.upgrader upgrader.go:216 no waiter, upgrader is done
2023-07-12 20:49:17 ERROR juju.worker.meterstatus runner.go:93 error running "meter-status-changed": charm missing from disk
2023-07-12 20:49:17 INFO juju.worker.uniter uniter.go:326 unit "postgresql-admin/0" started
2023-07-12 20:49:17 INFO juju.worker.uniter uniter.go:634 resuming charm install
2023-07-12 20:49:17 INFO juju.worker.uniter.charm bundles.go:78 downloading ch:amd64/jammy/data-integrator-11 from API server
2023-07-12 20:49:20 INFO juju.worker.uniter uniter.go:344 hooks are retried true
2023-07-12 20:49:20 INFO juju.worker.uniter.storage resolver.go:127 initial storage attachments ready
2023-07-12 20:49:20 INFO juju.worker.uniter resolver.go:159 found queued "install" hook
2023-07-12 20:49:20 INFO unit.postgresql-admin/0.juju-log server.go:316 Running legacy hooks/install.
2023-07-12 20:49:21 INFO juju.worker.uniter.operation runhook.go:159 ran "install" hook (via hook dispatching script: dispatch)
2023-07-12 20:49:21 INFO juju.worker.uniter.operation runhook.go:159 ran "data-integrator-peers-relation-created" hook (via hook dispatching script: dispatch)
2023-07-12 20:49:21 INFO juju.worker.uniter resolver.go:159 found queued "leader-elected" hook
2023-07-12 20:49:22 INFO juju.worker.uniter.operation runhook.go:159 ran "leader-elected" hook (via hook dispatching script: dispatch)
2023-07-12 20:49:22 INFO juju.worker.uniter.operation runhook.go:159 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:49:22 INFO juju.worker.uniter resolver.go:159 found queued "start" hook
2023-07-12 20:49:23 INFO unit.postgresql-admin/0.juju-log server.go:316 Running legacy hooks/start.
2023-07-12 20:49:23 INFO juju.worker.uniter.operation runhook.go:159 ran "start" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:18 INFO juju.worker.uniter.operation runhook.go:159 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:29 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-created" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:30 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-joined" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:32 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:50:34 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 20:53:34 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 20:59:00 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:04:12 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:08:24 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:14:12 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:14:28 INFO juju.worker.uniter.operation runhook.go:159 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-07-12 21:14:30 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 21:20:02 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:25:52 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:31:39 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:37:14 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:43:13 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:48:33 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:54:21 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 21:58:26 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:45 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-departed" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:46 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-broken" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:47 INFO juju.worker.uniter.operation runhook.go:159 ran "data-integrator-peers-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:47 INFO juju.worker.uniter.operation runhook.go:159 ran "data-integrator-peers-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 22:01:59 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-created" hook (via hook dispatching script: dispatch)
2023-07-12 22:02:01 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-joined" hook (via hook dispatching script: dispatch)
2023-07-12 22:02:02 INFO unit.postgresql-admin/0.juju-log server.go:316 postgresql:7: database created at 2023-07-12 22:02:02.859614
2023-07-12 22:02:03 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 22:02:04 INFO juju.worker.uniter.operation runhook.go:159 ran "data-integrator-peers-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 22:02:05 INFO juju.worker.uniter.operation runhook.go:159 ran "postgresql-relation-changed" hook (via hook dispatching script: dispatch)
2023-07-12 22:03:21 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 22:09:15 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)
2023-07-12 22:15:04 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via hook dispatching script: dispatch)

Additional context

Ref to another issue #182

github-actions[bot] commented 1 year ago

https://warthogs.atlassian.net/browse/DPE-2236

marceloneppel commented 1 year ago

Hi @gustavosr98! https://github.com/canonical/postgresql-k8s-operator/pull/201 will fix this error in the PG k8s charm and as soon as it's merged I'm gonna replicate the fix here.

marceloneppel commented 1 year ago

Fixed on https://github.com/canonical/postgresql-operator/pull/209. Revision 323 (latest on 14/edge channel has the fix).