Kafka topics missing when using external kafka

[zetaab]

I have installed sentry-kubernetes using AWS MSK

% kubectl get pods
NAME                                                              READY   STATUS    RESTARTS      AGE
keydb-0                                                           2/2     Running   0             3h18m
keydb-1                                                           2/2     Running   0             3h19m
keydb-2                                                           2/2     Running   0             3h19m
sentry-billing-metrics-consumer-c9bb54bb5-m99ld                   1/1     Running   0             29m
sentry-clickhouse-0                                               1/1     Running   0             70m
sentry-clickhouse-1                                               1/1     Running   0             70m
sentry-clickhouse-2                                               1/1     Running   0             70m
sentry-cron-9bcff65f-lkb6h                                        1/1     Running   0             70m
sentry-ingest-consumer-7d6789fc96-rljl6                           1/1     Running   2 (14s ago)   103s
sentry-ingest-metrics-consumer-perf-8456b8cf74-nnt7z              1/1     Running   2 (33s ago)   103s
sentry-ingest-metrics-consumer-rh-5c757b4c46-cswm9                1/1     Running   2 (33s ago)   103s
sentry-ingest-monitors-c888787c9-5n7hd                            1/1     Running   2 (34s ago)   103s
sentry-ingest-replay-recordings-7b878d5554-bddpp                  1/1     Running   2 (33s ago)   103s
sentry-memcached-5cc8644d8d-wwpqk                                 1/1     Running   0             51m
sentry-post-process-forward-errors-6ddcc5979f-wxv95               1/1     Running   0             29m
sentry-post-process-forward-transactions-5885f48b5-xgxxw          1/1     Running   0             29m
sentry-rabbitmq-0                                                 1/1     Running   0             70m
sentry-rabbitmq-1                                                 1/1     Running   0             68m
sentry-rabbitmq-2                                                 1/1     Running   0             67m
sentry-relay-68bbbc966d-7pjs7                                     1/1     Running   0             28m
sentry-relay-68bbbc966d-dhflf                                     1/1     Running   0             28m
sentry-relay-68bbbc966d-zbcdq                                     1/1     Running   0             28m
sentry-sessions-consumer-65cccd5456-x5k7s                         1/1     Running   0             28m
sentry-snuba-api-764c5d4dc6-6mm5d                                 1/1     Running   0             70m
sentry-snuba-api-764c5d4dc6-7cjkd                                 1/1     Running   0             70m
sentry-snuba-api-764c5d4dc6-vjq4p                                 1/1     Running   0             70m
sentry-snuba-consumer-79b9b44c79-qtp8m                            1/1     Running   0             29m
sentry-snuba-outcomes-consumer-c96c6cddd-4p477                    1/1     Running   0             28m
sentry-snuba-replacer-7f4869f47-wpdq6                             1/1     Running   0             28m
sentry-snuba-replays-consumer-5bbccdc647-bls4f                    1/1     Running   0             28m
sentry-snuba-subscription-consumer-events-758bdbbf6f-vw2c8        1/1     Running   0             28m
sentry-snuba-subscription-consumer-sessions-566499dfc-h6txs       1/1     Running   0             28m
sentry-snuba-subscription-consumer-transactions-57b6c88d772qwwh   1/1     Running   0             28m
sentry-snuba-transactions-consumer-5b4fcc9444-694rl               1/1     Running   0             28m
sentry-subscription-consumer-events-58b9c9698d-tlqjn              1/1     Running   0             28m
sentry-subscription-consumer-transactions-7f6d644448-6lj9t        1/1     Running   0             28m
sentry-symbolicator-api-8648478bd-cl4g4                           1/1     Running   0             70m
sentry-web-844ccc978d-9ppvj                                       1/1     Running   0             32m
sentry-web-844ccc978d-dgtsg                                       1/1     Running   0             32m
sentry-web-844ccc978d-fnd48                                       1/1     Running   0             42m
sentry-worker-5cc4d9c8d5-b4fx9                                    1/1     Running   0             42m
sentry-worker-5cc4d9c8d5-cnqks                                    1/1     Running   0             42m
sentry-worker-5cc4d9c8d5-pgqlj                                    1/1     Running   0             42m
sentry-zookeeper-clickhouse-0                                     1/1     Running   0             70m
sentry-zookeeper-clickhouse-1                                     1/1     Running   0             70m
sentry-zookeeper-clickhouse-2                                     1/1     Running   0             70m

when checking the logs it will say that these topics do have problems (pods in crashloop because of that):

ingest-attachments ingest-events ingest-metrics ingest-monitors ingest-performance-metrics ingest-replay-recordings ingest-transactions

example:

RuntimeError: Timeout when waiting for Kafka topic 'ingest-events' to become available, last error: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Broker: Unknown topic or partition"}

so for me this looks like some component has not created the needed topics or partitions?

the helm configuration:

kafka:
  enabled: false
externalKafka:
  - host: xxa.eu-central-1.amazonaws.com
    port: 9092
  - host: xxb.eu-central-1.amazonaws.com
    port: 9092
  - host: xxc.eu-central-1.amazonaws.com
    port: 9092

How I can create these topics manually, or what is the component that should do it?

[zetaab]

root@i-xx:~/kafka_2.13-3.4.0/bin# ./kafka-topics.sh --bootstrap-server xx.amazonaws.com:9092 --list
__amazon_msk_canary
__consumer_offsets
cdc
event-replacements
events
events-subscription-results
generic-events
generic-metrics-subscription-results
ingest-replay-events
ingest-sessions
metrics-subscription-results
outcomes
processed-profiles
profiles-call-tree
scheduled-subscriptions-events
scheduled-subscriptions-generic-metrics-counters
scheduled-subscriptions-generic-metrics-distributions
scheduled-subscriptions-generic-metrics-sets
scheduled-subscriptions-metrics
scheduled-subscriptions-sessions
scheduled-subscriptions-transactions
sessions-subscription-results
snuba-attribution
snuba-commit-log
snuba-dead-letter-generic-events
snuba-dead-letter-generic-metrics
snuba-dead-letter-metrics
snuba-dead-letter-querylog
snuba-dead-letter-replays
snuba-dead-letter-sessions
snuba-generic-events-commit-log
snuba-generic-metrics
snuba-generic-metrics-counters-commit-log
snuba-generic-metrics-distributions-commit-log
snuba-generic-metrics-sets-commit-log
snuba-metrics
snuba-metrics-commit-log
snuba-queries
snuba-sessions-commit-log
snuba-transactions-commit-log
transactions
transactions-subscription-results

so there are some topics but not all of them

[zetaab]

I created now these topics manually using

kafka-topics.sh --create --topic ingest-transactions --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-attachments --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-events --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-metrics --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-monitors --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-performance-metrics --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-replay-recordings --bootstrap-server foo.eu-central-1.amazonaws.com:9092

[stevefan1999-personal]

@Mokto @TartanLeGrand Can we reopen this? I actually got this problem again when I'm using Redpanda externally as it claims to be Kafka-compatible, and I'm giving it a try:

apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: sentry
  namespace: redpanda-operator
spec:
  chartRef:
    chartVersion: 5.8.13
  clusterSpec:
    statefulset:
      replicas: 1
    tls:
      enabled: false

This creates an external Kafka instance at sentry.redpanda-operator:9093 so I changed my values:

kafka:
  enabled: false
externalKafka:
  host: "sentry.redpanda-operator"
  port: 9093

There are still few topics not created, I redid the Kafka init job but I find no topics created as well

I created now these topics manually using

kafka-topics.sh --create --topic ingest-transactions --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-attachments --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-events --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-metrics --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-monitors --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-performance-metrics --bootstrap-server foo.eu-central-1.amazonaws.com:9092
kafka-topics.sh --create --topic ingest-replay-recordings --bootstrap-server foo.eu-central-1.amazonaws.com:9092

Almost exactly the same

[TartanLeGrand]

With the next version of Sentry => 24.6.0, sentry have add kafka migrations. So I don't think we need to reopen this issue. We'll just have to wait for the next version of the chart.