redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.14k stars 557 forks source link

CI Failure (key symptom) in `PartitionMovementTest.test_deletion_stops_move` #17362

Open vbotbuildovich opened 3 months ago

vbotbuildovich commented 3 months ago

https://buildkite.com/redpanda/redpanda/builds/46692

Module: rptest.tests.partition_movement_test
Class: PartitionMovementTest
Method: test_deletion_stops_move
Arguments: {
    "num_to_upgrade": 2
}
test_id:    PartitionMovementTest.test_deletion_stops_move
status:     FAIL
run time:   27.566 seconds

Exception('KafProducer-0-140626338786576-worker-1: Traceback (most recent call last):\n  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/kaf_producer.py", line 32, in _worker\n    for line in out_iter:\n  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 687, in next\n    return next(self.iter_obj)\n  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 354, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-19: Command \'echo $$ ; for (( i=0; i < 1000; i++ )) ; do export KEY=key-$(printf %08d $i) ; export VALUE=record-$(printf %08d $i) ; echo $VALUE | kaf produce -b docker-rp-6:9092,docker-rp-7:9092,docker-rp-24:9092 --key $KEY test-topic ; done\' returned non-zero exit status 1.\n')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 104, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/partition_movement_test.py", line 598, in test_deletion_stops_move
    producer.wait()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 74, in wait
    self._propagate_exceptions()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 100, in _propagate_exceptions
    raise Exception(self.errors)
Exception: KafProducer-0-140626338786576-worker-1: Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker
    self._worker(idx, node)
  File "/root/tests/rptest/services/kaf_producer.py", line 32, in _worker
    for line in out_iter:
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 687, in next
    return next(self.iter_obj)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 354, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-19: Command 'echo $$ ; for (( i=0; i < 1000; i++ )) ; do export KEY=key-$(printf %08d $i) ; export VALUE=record-$(printf %08d $i) ; echo $VALUE | kaf produce -b docker-rp-6:9092,docker-rp-7:9092,docker-rp-24:9092 --key $KEY test-topic ; done' returned non-zero exit status 1.

JIRA Link: CORE-1922

vbotbuildovich commented 3 months ago

*https://buildkite.com/redpanda/vtools/builds/12538