search

canonical / postgresql-test-app

Test app for Charmed PostgreSQL K8s/VM
https://charmhub.io/postgresql-test-app
Apache License 2.0
1 stars 0 forks source link

Problem running tests for postgresql_operator (rev103) #119

Closed BalabaDmitri closed 2 months ago

BalabaDmitri commented 2 months ago

Steps to reproduce

  1. tox -e integration

Expected behavior

The integration tests finishes successfully.

Actual behavior

When deploying, the default is the latest revision (rev103), and then the tests fail.

await ops_test.model.deploy(
    "postgresql-test-app",
     application_name="postgresql-test-app",
     series="jammy",
     channel="edge",
)

Errors in tests:

  1. test_full_cluster_restart
  2. test_network_cut
  3. test_network_cut_without_ip_change

    Versions

Operating system: Ubuntu 22.04.4 LTS

Juju CLI: 3.1.8-genericlinux-amd64

Juju agent: 3.1.7

Charm revision: 103

LXD: 5.21.1 LTS

Log output

Juju debug log: log for test test_full_cluster_restart

unit-postgresql-1: 19:12:50 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-test-app-0: 19:12:51 INFO unit.postgresql-test-app/0.juju-log first-database:4: database created at 2024-04-16 16:12:51.360262
unit-postgresql-test-app-0: 19:12:51 INFO unit.postgresql-test-app/0.juju-log first-database:4: first database credentials: relation-4 VLOsPemhPVKrQC6G
unit-postgresql-test-app-0: 19:12:51 INFO juju.worker.uniter.operation ran "first-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-1: 19:12:52 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch)
unit-postgresql-2: 19:12:54 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 19:12:54 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-1: 19:12:54 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-test-app-0: 19:13:12 INFO juju.worker.uniter.operation ran "postgresql-test-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 19:13:34 INFO unit.postgresql/0.juju-log Cluster topology changed
unit-postgresql-test-app-0: 19:13:36 INFO juju.worker.uniter.operation ran "postgresql-test-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-1: 19:13:38 INFO unit.postgresql/1.juju-log Cluster topology changed
unit-postgresql-2: 19:13:40 INFO unit.postgresql/2.juju-log Cluster topology changed
unit-postgresql-test-app-0: 19:13:52 INFO juju.worker.uniter.operation ran "postgresql-test-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-test-app-0: 19:15:17 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:19 INFO juju.worker.uniter.operation ran "postgresql-test-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-test-app-0: 19:15:22 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:26 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:30 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:34 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:38 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:42 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:46 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:51 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:55 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:15:59 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:16:03 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:16:07 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:16:11 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:16:15 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:16:19 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:16:24 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-test-app-0: 19:16:28 ERROR unit.postgresql-test-app/0.juju-log Unable to drop table
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-test-app-0/charm/./src/charm.py", line 284, in _on_clear_continuous_writes_action
    cursor.execute("DROP TABLE IF EXISTS continuous_writes;")
psycopg2.errors.ReadOnlySqlTransaction: cannot execute DROP TABLE in a read-only transaction

unit-postgresql-2: 19:16:28 INFO unit.postgresql/2.juju-log Starting cluster topology observer process
unit-postgresql-2: 19:16:28 INFO unit.postgresql/2.juju-log Started cluster topology observer process with PID 23184
unit-postgresql-2: 19:16:29 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-postgresql-1: 19:16:30 INFO unit.postgresql/1.juju-log database-peers:0: Starting cluster topology observer process
unit-postgresql-0: 19:16:30 INFO unit.postgresql/0.juju-log database-peers:0: Starting cluster topology observer process
unit-postgresql-1: 19:16:30 INFO unit.postgresql/1.juju-log database-peers:0: Started cluster topology observer process with PID 30764
unit-postgresql-0: 19:16:30 INFO unit.postgresql/0.juju-log database-peers:0: Started cluster topology observer process with PID 31524
unit-postgresql-1: 19:16:30 INFO unit.postgresql/1.juju-log database-peers:0: Reconfiguring cluster
unit-postgresql-1: 19:16:31 INFO unit.postgresql/1.juju-log database-peers:0: not all members are ready

Additional context

When I change revision to rev101 the tests pass successfully

await ops_test.model.deploy(
    "postgresql-test-app",
     application_name="postgresql-test-app",
     series="jammy",
     channel="edge",
     revision=101,
)
dragomirp commented 2 months ago

Hi, there should be no functional difference between 101 and 103: https://github.com/canonical/postgresql-test-app/compare/rev101...rev103

It's an update of a linting dependency and Juju agent version used in tests.

If it happens again, please check if there's a primary unit in the Patroni's cluster endpoint

taurus-forever commented 2 months ago

@BalabaDmitri it sounds like Raft issue on PostgreSQL VM charm. We are planning to release the next stable version of test application (rev 110+111). Is it still reproducible, affecting you? Tnx!

BalabaDmitri commented 2 months ago

@taurus-forever I ran the test several times and it worked successfully. Everything is working

taurus-forever commented 2 months ago

I am going to resolve this ticket as the error message looks related to raft library issue (test-app is trying to connect read-only PostgreSQL unit). It is not related to postgresql-test-app, IMHO. Feel free to reopen if the issue is reproducible.

P.S. we are planning to promote the next stable and no issues noticed there (rev 117).