search

sentry-kubernetes / charts

Easily deploy Sentry on your Kubernetes Cluster
MIT License
1.09k stars 514 forks source link

Why does sentry-issue-occurrence-consumer can't connect to sentry-kafka-controller？ #1597

Open ZLBillShaw opened 5 hours ago

ZLBillShaw commented 5 hours ago

I configured an external ClickHouse, and logs show it has connected successfully. However, it seems that the consumer cannot connect to Kafka. My Kafka setup is unchanged from the default in values.yaml. Below are the logs from sentry-issue-occurrence-consumerpod

2024-11-06 04:25:35,434 Initializing Snuba...
2024-11-06 04:25:37,340 Snuba initialization took 1.9061147570610046s
2024-11-06 04:25:37,349 Consumer Starting
2024-11-06 04:25:37,350 Checking Clickhouse connections...
2024-11-06 04:25:37,482 Successfully connected to Clickhouse: cluster_name=None
2024-11-06 04:25:37,483 librdkafka log level: 6
2024-11-06 04:25:40,498 New partitions assigned: {Partition(topic=Topic(name='generic-events'), index=0): 0}
%6|1730871000.715|FAIL|rdkafka#consumer-3| [thrd:sentry-kafka-controller-1.sentry-kafka-controller-headless.defa]: sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Disconnected (after 3860218ms in state UP)
2024-11-06 05:30:00,716 Error callback from librdKafka -195, _TRANSPORT, sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Disconnected (after 3860218ms in state UP)
%3|1730871000.717|FAIL|rdkafka#consumer-3| [thrd:sentry-kafka-controller-1.sentry-kafka-controller-headless.defa]: sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 1ms in state CONNECT)
2024-11-06 05:30:00,718 Error callback from librdKafka -195, _TRANSPORT, sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 1ms in state CONNECT)
%3|1730871000.984|FAIL|rdkafka#consumer-3| [thrd:sentry-kafka-controller-1.sentry-kafka-controller-headless.defa]: sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2024-11-06 05:30:00,984 Error callback from librdKafka -195, _TRANSPORT, sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1730871001.501|FAIL|rdkafka#producer-1| [thrd:sentry-kafka-controller-1.sentry-kafka-controller-headless.defa]: sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 0ms in state CONNECT)
%6|1730873082.680|FAIL|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Disconnected (after 5945189ms in state UP)
2024-11-06 06:04:42,681 Error callback from librdKafka -195, _TRANSPORT, GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Disconnected (after 5945189ms in state UP)
%3|1730873082.681|FAIL|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Connect to ipv4#11.157.248.126:9092 failed: Connection refused (after 0ms in state CONNECT)
2024-11-06 06:04:42,683 Error callback from librdKafka -195, _TRANSPORT, GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Connect to ipv4#11.157.248.126:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1730873082.871|FAIL|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Connect to ipv4#11.157.248.126:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2024-11-06 06:04:42,871 Error callback from librdKafka -195, _TRANSPORT, GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Connect to ipv4#11.157.248.126:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)