search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.14k stars 558 forks source link

CI Failure (unable to create topic) in `TestReadReplicaService.test_writes_forbidden` #18800

Open vbotbuildovich opened 3 weeks ago

vbotbuildovich commented 3 weeks ago

https://buildkite.com/redpanda/vtools/builds/14278

Module: rptest.tests.read_replica_e2e_test
Class: TestReadReplicaService
Method: test_writes_forbidden
Arguments: {
    "partition_count": 10,
    "cloud_storage_type": 1
}
test_id:    TestReadReplicaService.test_writes_forbidden
status:     FAIL
run time:   26.230 seconds

RpkException('command /opt/redpanda/bin/rpk topic -X brokers=ducktape-node-19-manually-casual-sheepdog:9092,ducktape-node-20-manually-casual-sheepdog:9092,ducktape-node-5-manually-casual-sheepdog:9092 create panda-topic --partitions 1 --topic-config redpanda.remote.readreplica:panda-bucket-e041a782-2197-11ef-a293-a16a8c6db7d0 -v returned 1, output: TOPIC        STATUS\npanda-topic  UNKNOWN_SERVER_ERROR: Unable to perform requested topic operation \n', '10:56:21.562  DEBUG  opening connection to broker  {"addr": "ducktape-node-19-manually-casual-sheepdog:9092", "broker": "seed_0"}\n10:56:21.563  DEBUG  connection opened to broker  {"addr": "ducktape-node-19-manually-casual-sheepdog:9092", "broker": "seed_0"}\n10:56:21.563  DEBUG  issuing api versions request  {"broker": "seed_0", "version": 3}\n10:56:21.563  DEBUG  wrote ApiVersions v3  {"broker": "seed_0", "bytes_written": 31, "write_wait": "33.325µs", "time_to_write": "20.172µs", "err": null}\n10:56:21.563  DEBUG  read ApiVersions v3  {"broker": "seed_0", "bytes_read": 310, "read_wait": "32.355µs", "time_to_read": "280.065µs", "err": null}\n10:56:21.563  DEBUG  connection initialized successfully  {"addr": "ducktape-node-19-manually-casual-sheepdog:9092", "broker": "seed_0"}\n10:56:21.563  DEBUG  wrote Metadata v7  {"broker": "seed_0", "bytes_written": 22, "write_wait": "1.418259ms", "time_to_write": "22.026µs", "err": null}\n10:56:21.564  DEBUG  read Metadata v7  {"broker": "seed_0", "bytes_read": 229, "read_wait": "49.127µs", "time_to_read": "135.874µs", "err": null}\n10:56:21.564  DEBUG  opening connection to broker  {"addr": "ducktape-node-5-manually-casual-sheepdog:9092", "broker": "1"}\n10:56:21.564  DEBUG  connection opened to broker  {"addr": "ducktape-node-5-manually-casual-sheepdog:9092", "broker": "1"}\n10:56:21.564  DEBUG  issuing api versions request  {"broker": "1", "version": 3}\n10:56:21.564  DEBUG  wrote ApiVersions v3  {"broker": "1", "bytes_written": 31, "write_wait": "18.533µs", "time_to_write": "18.484µs", "err": null}\n10:56:21.564  DEBUG  read ApiVersions v3  {"broker": "1", "bytes_read": 310, "read_wait": "25.179µs", "time_to_read": "266.166µs", "err": null}\n10:56:21.564  DEBUG  connection initialized successfully  {"addr": "ducktape-node-5-manually-casual-sheepdog:9092", "broker": "1"}\n10:56:21.564  DEBUG  wrote CreateTopics v6  {"broker": "1", "bytes_written": 125, "write_wait": "894.606µs", "time_to_write": "9.532µs", "err": null}\n10:56:21.641  DEBUG  read CreateTopics v6  {"broker": "1", "bytes_read": 86, "read_wait": "31.472µs", "time_to_read": "76.085628ms", "err": null}\n10:56:21.641  DEBUG  redpanda returned error message: Unable to perform requested topic operation \n', 1, 'TOPIC        STATUS\npanda-topic  UNKNOWN_SERVER_ERROR: Unable to perform requested topic operation \n')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/read_replica_e2e_test.py", line 384, in test_writes_forbidden
    self._setup_read_replica(partition_count=partition_count,
  File "/home/ubuntu/redpanda/tests/rptest/tests/read_replica_e2e_test.py", line 242, in _setup_read_replica
    wait_until(
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 53, in wait_until
    raise e
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 44, in wait_until
    if condition():
  File "/home/ubuntu/redpanda/tests/rptest/tests/read_replica_e2e_test.py", line 199, in create_read_replica_topic_success
    self.create_read_replica_topic()
  File "/home/ubuntu/redpanda/tests/rptest/tests/read_replica_e2e_test.py", line 172, in create_read_replica_topic
    create_read_replica_topic(self.second_cluster, self.topic_name,
  File "/home/ubuntu/redpanda/tests/rptest/tests/read_replica_e2e_test.py", line 110, in create_read_replica_topic
    rpk_dst_cluster.create_topic(topic_name, config=conf)
  File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 355, in create_topic
    wait_until_result(create_topic,
  File "/home/ubuntu/redpanda/tests/rptest/util.py", line 94, in wait_until_result
    wait_until(wrapped_condition, *args, **kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 53, in wait_until
    raise e
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 44, in wait_until
    if condition():
  File "/home/ubuntu/redpanda/tests/rptest/util.py", line 81, in wrapped_condition
    cond = condition()
rptest.clients.rpk.RpkException: RpkException<command /opt/redpanda/bin/rpk topic -X brokers=ducktape-node-19-manually-casual-sheepdog:9092,ducktape-node-20-manually-casual-sheepdog:9092,ducktape-node-5-manually-casual-sheepdog:9092 create panda-topic --partitions 1 --topic-config redpanda.remote.readreplica:panda-bucket-e041a782-2197-11ef-a293-a16a8c6db7d0 -v returned 1, output: TOPIC        STATUS
panda-topic  UNKNOWN_SERVER_ERROR: Unable to perform requested topic operation 
; stderr: 10:56:21.562  DEBUG  opening connection to broker  {"addr": "ducktape-node-19-manually-casual-sheepdog:9092", "broker": "seed_0"}
10:56:21.563  DEBUG  connection opened to broker  {"addr": "ducktape-node-19-manually-casual-sheepdog:9092", "broker": "seed_0"}
10:56:21.563  DEBUG  issuing api versions request  {"broker": "seed_0", "version": 3}
10:56:21.563  DEBUG  wrote ApiVersions v3  {"broker": "seed_0", "bytes_written": 31, "write_wait": "33.325µs", "time_to_write": "20.172µs", "err": null}
10:56:21.563  DEBUG  read ApiVersions v3  {"broker": "seed_0", "bytes_read": 310, "read_wait": "32.355µs", "time_to_read": "280.065µs", "err": null}
10:56:21.563  DEBUG  connection initialized successfully  {"addr": "ducktape-node-19-manually-casual-sheepdog:9092", "broker": "seed_0"}
10:56:21.563  DEBUG  wrote Metadata v7  {"broker": "seed_0", "bytes_written": 22, "write_wait": "1.418259ms", "time_to_write": "22.026µs", "err": null}
10:56:21.564  DEBUG  read Metadata v7  {"broker": "seed_0", "bytes_read": 229, "read_wait": "49.127µs", "time_to_read": "135.874µs", "err": null}
10:56:21.564  DEBUG  opening connection to broker  {"addr": "ducktape-node-5-manually-casual-sheepdog:9092", "broker": "1"}
10:56:21.564  DEBUG  connection opened to broker  {"addr": "ducktape-node-5-manually-casual-sheepdog:9092", "broker": "1"}
10:56:21.564  DEBUG  issuing api versions request  {"broker": "1", "version": 3}
10:56:21.564  DEBUG  wrote ApiVersions v3  {"broker": "1", "bytes_written": 31, "write_wait": "18.533µs", "time_to_write": "18.484µs", "err": null}
10:56:21.564  DEBUG  read ApiVersions v3  {"broker": "1", "bytes_read": 310, "read_wait": "25.179µs", "time_to_read": "266.166µs", "err": null}
10:56:21.564  DEBUG  connection initialized successfully  {"addr": "ducktape-node-5-manually-casual-sheepdog:9092", "broker": "1"}
10:56:21.564  DEBUG  wrote CreateTopics v6  {"broker": "1", "bytes_written": 125, "write_wait": "894.606µs", "time_to_write": "9.532µs", "err": null}
10:56:21.641  DEBUG  read CreateTopics v6  {"broker": "1", "bytes_read": 86, "read_wait": "31.472µs", "time_to_read": "76.085628ms", "err": null}
10:56:21.641  DEBUG  redpanda returned error message: Unable to perform requested topic operation 
; stdout: TOPIC        STATUS
panda-topic  UNKNOWN_SERVER_ERROR: Unable to perform requested topic operation 
; returncode: 1>

JIRA Link: CORE-3208

vbotbuildovich commented 6 days ago

*https://buildkite.com/redpanda/vtools/builds/15023