search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.17k stars 561 forks source link

CI Failure (Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe)) in `RandomNodeOperationsTest.test_node_operations` #16468

Closed vbotbuildovich closed 3 weeks ago

vbotbuildovich commented 5 months ago

https://buildkite.com/redpanda/vtools/builds/11696

Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
    "num_to_upgrade": 0,
    "enable_failures": false,
    "with_tiered_storage": true
}
test_id:    RandomNodeOperationsTest.test_node_operations
status:     FAIL
run time:   1984.848 seconds

<BadLogLines nodes=ip-172-31-33-31(1) example="ERROR 2024-02-02 22:02:37,251 [shard 0:au  ] s3 - util.cc:105 - Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::__nested<std::__1::system_error> (error GnuTLS:-53, Error in the push function.): std::__1::system_error (error system:32, sendmsg: Broken pipe))">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 168, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1413, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-33-31(1) example="ERROR 2024-02-02 22:02:37,251 [shard 0:au  ] s3 - util.cc:105 - Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::__nested<std::__1::system_error> (error GnuTLS:-53, Error in the push function.): std::__1::system_error (error system:32, sendmsg: Broken pipe))">

JIRA Link: CORE-1754

travisdowns commented 5 months ago

In handle_client_transport_error where we categorize the error we already handle GnuTLS push error, we probably need to also handle the nested exception case as shown here. Have to check the seastar code but I guess while unwinding during the original GnuTLS exception we try to send more data down the pipe and get broken pipe (if this occurs or not in some run may be ~random depending on the exact state of the socket buffer).

vbotbuildovich commented 3 months ago

*https://buildkite.com/redpanda/vtools/builds/12295

piyushredpanda commented 3 weeks ago

Not seen in at 6 weeks, closing