search

MaterializeInc / materialize

The Cloud Operational Data Store: use SQL to transform, deliver, and act on fast-changing data.
https://materialize.com
Other
5.71k stars 465 forks source link

pg-cdc silent-connection-drop is flaky: AssertionError: Expected one replication slot but found 2 slots #28294

Open def- opened 1 month ago

def- commented 1 month ago

What version of Materialize are you using?

44a034698ce8

What is the issue?

Seen in https://buildkite.com/materialize/test/builds/86052#0190bd3b-287c-4dca-8909-1c686aafc4c8

==> mzcompose: test case silent-connection-drop failed: builtins.AssertionError: Expected one replication slot but found 2 slots
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-00d0b8e7fafd132b8-1/materialize/test/misc/python/materialize/mzcompose/composition.py", line 578, in test_case
    yield
  File "/var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-00d0b8e7fafd132b8-1/materialize/test/test/pg-cdc/mzcompose.py", line 319, in workflow_default
    c.workflow(name)
  File "/var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-00d0b8e7fafd132b8-1/materialize/test/misc/python/materialize/mzcompose/composition.py", line 460, in workflow
    func(self, parser)
  File "/var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-00d0b8e7fafd132b8-1/materialize/test/test/pg-cdc/mzcompose.py", line 197, in workflow_silent_connection_drop
    _verify_only_one_replication_slot_exists(pg_conn)
  File "/var/lib/buildkite-agent/builds/buildkite-aarch64-small-d306b64-i-00d0b8e7fafd132b8-1/materialize/test/test/pg-cdc/mzcompose.py", line 260, in _verify_only_one_replication_slot_exists
    count_slots == 1
AssertionError: Expected one replication slot but found 2 slots

@nrainer-materialize You added this test recently, so I assume it's a test issue: https://github.com/MaterializeInc/materialize/pull/27727 ci-regexp: builtins.AssertionError: Expected one replication slot but found 2 slots

nrainer-materialize commented 1 month ago

This is a bit surprising.

I added a further assertion with https://github.com/MaterializeInc/materialize/pull/28302.

@rjobanp, any ideas why test/pg-cdc/override/silent-connection-drop-part-1.td could (sometimes) open two replication slots?