search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.69k stars 591 forks source link

CI Failure (go-kafka-serde returned non-zero exit status 1) in `SchemaRegistryAutoAuthTest.test_serde_client` #10001

Closed ztlpn closed 1 year ago

ztlpn commented 1 year ago

https://buildkite.com/redpanda/redpanda/builds/26840#01877093-ce14-4e23-8fd7-91498e58fb0e

This is v22.3.x and a PR build (but I think the error is unrelated)

Module: rptest.tests.schema_registry_test
Class:  SchemaRegistryAutoAuthTest
Method: test_serde_client
Arguments:
{
  "client_type": 3,
  "protocol": 2
}
test_id:    rptest.tests.schema_registry_test.SchemaRegistryAutoAuthTest.test_serde_client.protocol=PROTOBUF.client_type=Golang
status:     FAIL
run time:   14.731 seconds

    Exception('SerdeClient-0-140510478558704-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/serde_client.py", line 93, in _worker\n    for line in ssh_output:\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-9: Command \'/opt/redpanda-tests/go/go-kafka-serde/go-kafka-serde --brokers docker-rp-13:9092,docker-rp-21:9092,docker-rp-22:9092 --topic serde-topic-PROTOBUF-Golang --schema-registry http://docker-rp-21:8081 --consumer-group bf4ddacc-5d5c-4130-a0a6-99bbc9ab6ed7 --protocol PROTOBUF --count 2 --debug --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/schema_registry_test.py", line 1162, in test_serde_client
    client.wait()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 74, in wait
    self._propagate_exceptions()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 100, in _propagate_exceptions
    raise Exception(self.errors)
Exception: SerdeClient-0-140510478558704-worker-1: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker
    self._worker(idx, node)
  File "/root/tests/rptest/services/serde_client.py", line 93, in _worker
    for line in ssh_output:
  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 658, in next
    return next(self.iter_obj)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-9: Command '/opt/redpanda-tests/go/go-kafka-serde/go-kafka-serde --brokers docker-rp-13:9092,docker-rp-21:9092,docker-rp-22:9092 --topic serde-topic-PROTOBUF-Golang --schema-registry http://docker-rp-21:8081 --consumer-group bf4ddacc-5d5c-4130-a0a6-99bbc9ab6ed7 --protocol PROTOBUF --count 2 --debug --security '{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}'' returned non-zero exit status 1.
ztlpn commented 1 year ago

@michael-redpanda can you take a look? Looks like the test is quite new.

ztlpn commented 1 year ago

looking at the test code, how do we guarantee that we've processed all producer events before checking numDelivered?

michael-redpanda commented 1 year ago

looking at the test code, how do we guarantee that we've processed all producer events before checking numDelivered?

I was just looking at that. Yeah it's probably that. I'll fix it in dev and the backport it to.

dotnwat commented 1 year ago

https://buildkite.com/redpanda/redpanda/builds/27280#01878de1-0dcc-48a3-9dd5-5791230f6783

dotnwat commented 1 year ago

https://buildkite.com/redpanda/redpanda/builds/27464#01879823-d2b1-494d-a763-55f5f85954be

dotnwat commented 1 year ago 
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=AVRO.client_type=Golang (1/35 runs)
  failure at 2023-04-20T06:48:04.201Z: Exception('SerdeClient-0-140338125742112-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-23: Command \'/opt/redpanda-tests/go/go-kafka-serde/go-kafka-serde --brokers docker-rp-15:9092,docker-rp-18:9092,docker-rp-12:9092 --topic serde-topic-AVRO-Golang --schema-registry http://docker-rp-18:8081 --consumer-group bd59515e-c5cb-4f56-b581-5d03c4571a61 --protocol AVRO --count 2 --debug --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/27568#01879d32-69b6-4002-897e-8e5cf251a18f
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=AVRO.client_type=Python (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566306121296-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-12-222: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-12-52:9092,ip-172-31-15-8:9092,ip-172-31-8-225:9092 --topic serde-topic-AVRO-Python --schema-registry http://ip-172-31-8-225:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol AVRO --count 2 --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=PROTOBUF.client_type=Golang (2/35 runs)
  failure at 2023-04-19T16:34:16.694Z: Exception('SerdeClient-0-139990264732912-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-12: Command \'/opt/redpanda-tests/go/go-kafka-serde/go-kafka-serde --brokers docker-rp-23:9092,docker-rp-11:9092,docker-rp-24:9092 --topic serde-topic-PROTOBUF-Golang --schema-registry http://docker-rp-23:8081 --consumer-group f15559ad-227c-487c-a112-a896cba33540 --protocol PROTOBUF --count 2 --debug --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/27503#01879a11-f89d-4bf0-bd3c-cc634598500c
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=PROTOBUF.client_type=Python.skip_known_types=False (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566064571888-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-2-84: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-12-52:9092,ip-172-31-15-8:9092,ip-172-31-12-222:9092 --topic serde-topic-PROTOBUF-Python --schema-registry http://ip-172-31-12-52:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol PROTOBUF --count 2 --skip-known-types --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryAutoAuthTest.test_serde_client.protocol=PROTOBUF.client_type=Python.skip_known_types=True (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566065297200-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-12-222: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-15-8:9092,ip-172-31-12-52:9092,ip-172-31-8-225:9092 --topic serde-topic-PROTOBUF-Python --schema-registry http://ip-172-31-8-225:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol PROTOBUF --count 2 --skip-known-types --security \'{"security_protocol": "SASL_PLAINTEXT", "sasl_mechanism": "SCRAM-SHA-256", "sasl_plain_username": "admin", "sasl_plain_password": "admin", "request_timeout_ms": 30000, "api_version_auto_timeout_ms": 3000}\'\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryTest.test_serde_client.protocol=AVRO.client_type=Python (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566064243824-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-2-84: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-15-8:9092,ip-172-31-12-52:9092,ip-172-31-12-222:9092 --topic serde-topic-AVRO-Python --schema-registry http://ip-172-31-12-52:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol AVRO --count 2\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryTest.test_serde_client.protocol=PROTOBUF.client_type=Python.skip_known_types=False (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566065676688-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-8-225: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-2-84:9092,ip-172-31-15-8:9092,ip-172-31-12-222:9092 --topic serde-topic-PROTOBUF-Python --schema-registry http://ip-172-31-15-8:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol PROTOBUF --count 2 --skip-known-types\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
FAIL test: SchemaRegistryTest.test_serde_client.protocol=PROTOBUF.client_type=Python.skip_known_types=True (2/35 runs)
  failure at 2023-04-20T16:36:08.166Z: Exception('SerdeClient-0-140566306121296-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/serde_client.py", line 101, in _worker\n    for line in ssh_output:\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 658, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 334, in output_generator\n    raise RemoteCommandError(self, cmd, exit_status, stderr.read())\nducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-2-84: Command \'python3 /tmp/python_librdkafka_serde_client.py --brokers ip-172-31-12-222:9092,ip-172-31-7-239:9092,ip-172-31-15-8:9092 --topic serde-topic-PROTOBUF-Python --schema-registry http://ip-172-31-7-239:8081 --consumer-group b4aec6fa-f8b8-465f-8a28-b4f2976d3d6c --protocol PROTOBUF --count 2 --skip-known-types\' returned non-zero exit status 1.\n')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7207#01879db3-1bee-43a6-b959-3230f72d5247
michael-redpanda commented 1 year ago

The failures in golang are known (and being addressed in #10184).

The python ones are due to an ImportError:

│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]: Traceback (most recent call last):                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]:   File "/tmp/python_librdkafka_serde_client.py", line 27, in <module>                                                                                                                                                                                                           │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]:     from payload_pb2 import Payload as ProtobufPayloadClass                                                                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]:   File "/tmp/payload_pb2.py", line 5, in <module>                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]:     from google.protobuf.internal import builder as _builder                                                                                                                                                                                                                    │
│                                                                                                                                                                                                                                                                                                                                                         │
│[DEBUG - 2023-04-20 13:12:33,292 - serde_client - _worker - lineno:102]: ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/usr/local/lib/python3.10/dist-packages/google/protobuf/internal/__init__.py)
michael-redpanda commented 1 year ago

I kicked off a cdt arm on #10184 and did not see those python errors.