External Kafka for sentry

[therus000]

Good day what the way to use an external kafka for sentry

externalKafka:
  ## Hostname or ip address of external kafka
  ##
  host: "cluster-01-kafka"
  port: 9092
  user: user
  password: password

i tried to used this configs. but it now work i wonder why is there is no setting for topic and it user can anyone help with this question?

[brianchan661]

i am using internal but did you try to set asHook to true, so it will initial the topics?

[MrRuban]

@therus000 Hi, I had the same problem. I solved it in the following way

snuba:
  dbInitJob:
    env:
    - name: KAFKA_SECURITY_PROTOCOL
      value: SASL_PLAINTEXT
    - name: KAFKA_SASL_MECHANISM
      value: SCRAM-SHA-512
    - name: KAFKA_SASL_USERNAME
      value: user
    - name: KAFKA_SASL_PASSWORD
      value: "password"

Remember that the user must have permissions. I use Yandex Cloud, the terraform for it looks like this:

resource "yandex_mdb_kafka_cluster" "sentry_kafka_dev" {
  name        = "sentry-kafka-dev"
  environment = "PRESTABLE"
  description = "Kafka dev cluster for Sentry stack"
  network_id  = data.yandex_vpc_network.dev.network_id

  subnet_ids = [data.yandex_vpc_subnet.dev_1b.subnet_id, ]

  config {
    version          = "3.3"
    brokers_count    = 1
    zones            = ["ru-central1-b"]
    assign_public_ip = false
    unmanaged_topics = true     # Important
    schema_registry  = false
    kafka {
      resources {
        resource_preset_id = "b3-c1-m4"
        disk_type_id       = "network-ssd"
        disk_size          = 60
      }
      kafka_config {
        auto_create_topics_enable = true     # Important
        message_max_bytes         = "50000000"     # Important
      }
    }
  }
}

resource "yandex_mdb_kafka_user" "user_events" {
  cluster_id = yandex_mdb_kafka_cluster.sentry_kafka_dev.id
  name       = user
  password   = "password"
  permission {
    role       = "ACCESS_ROLE_ADMIN"     # Important
    topic_name = "*"     # Important
  }
}

[therus000]

Thanx for answer, but I didn't understand, where the name of topic is fixed?

[MrRuban]

The topics will be created automatically. But you need that the user has permissions, and in the configuration of Kafka enabled auto-create topics

If there is a place where you can influence the creation of topics, it is in the field:

snuba:
  dbInitJob:

Good Luck

[zetaab]

there are also other components that will connect to kafka. All components needs similar settings. Sentry itself supports these https://github.com/getsentry/sentry/blob/128958855caffd9828cd9b1ef2cab0b6372bba64/src/sentry/utils/kafka_config.py#L5-L30 but these are configurable in this helm chart.

or perhaps its possible to override whole section https://github.com/sentry-kubernetes/charts/blob/develop/sentry/templates/configmap-sentry.yaml#L237-L246 with https://github.com/sentry-kubernetes/charts/blob/develop/sentry/templates/configmap-sentry.yaml#L518

[Mokto]

This issue is stale because it has been open for 30 days with no activity.

[Mokto]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[kashtan404]

@MrRuban Could you describe your part of configs for Sentry? I'm trying to set it with helm chart (official one but with auth options added). With bundled kafka everything ok, but with managed (yandex cloud) I got errors. with config

    DEFAULT_KAFKA_OPTIONS = {
        "bootstrap.servers": "rc1a-***.mdb.yandexcloud.net:9091",
        "ssl.ca.location": "/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt",
        "sasl.mechanism": "SCRAM-SHA-512",
        "security.protocol": "SASL_SSL",
        "sasl.username": "***",
        "sasl.password": "***",
        "message.max.bytes": 50000000,
        "socket.timeout.ms": 1000,
    }

%6|1718810315.260|FAIL|rdkafka#producer-1| [thrd:rc1a-***..mdb.yandexcloud.net:9091/bootstrap]: rc1a-***..mdb.yandexcloud.net:9091/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY, 5 identical error(s) suppressed)

and with config

    DEFAULT_KAFKA_OPTIONS = {
        "bootstrap.servers": "rc1a-***.mdb.yandexcloud.net:9092",
        "sasl.mechanism": "SCRAM-SHA-512",
        "security.protocol": "SASL_PLAINTEXT",
        "sasl.username": "***",
        "sasl.password": "***",
        "message.max.bytes": 50000000,
        "socket.timeout.ms": 1000,
    }

%4\|1718806836.418\|FAIL\|rdkafka#consumer-1\| \[thrd:rc1a-***.mdb.yandexcloud.net:9092/bootstrap\]: [rc1a-***..mdb.yandexcloud.net:9092/bootstrap:](http://rc1a-***..mdb.yandexcloud.net:9092/bootstrap:) Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 315ms in state UP, 4 identical error(s) suppressed)

I'm standing on the line to bootstrap general VM and install kafka on it.

[MrRuban]

@kashtan404 Hi! I finally realised that only the snuba component can connect in the way I described above, but the sentry component cannot. So I found Kafka-proxy, wrote a helm chart for it and stole the "topics-provisioning" job from bitnami kafka helm chart.

Titles of topics are taken from here

helm chart archive attached kafka-proxy.zip

[gwyn-bl]

@kashtan404 Hi, have you find how to connect to external kafka and resolve these errors?

[paulDashkevich]

@kashtan404 use kafka-proxy (single/multi-brokers) But I use managed clickhouse/Redis/Postgres/k8s/Kafka (Yandex Cloud too). Recently get loosing any transactions (errors/attachments/transactions) and this often happends. Do you have any stability in use sentry?

[MrRuban]

@kashtan404 Hello! Based on my personal attempts, I was able to achieve stable work only on ClickHouse version 21.8. Yandex Cloud does not provide such versions. I installed the right version of ClickHouse with the help of ansible, and Sentry started working much better.

[nadecancode]

@kashtan404 Hi! I finally realised that only the snuba component can connect in the way I described above, but the sentry component cannot. So I found Kafka-proxy, wrote a helm chart for it and stole the "topics-provisioning" job from bitnami kafka helm chart.

Titles of topics are taken from here

helm chart archive attached kafka-proxy.zip

Hello - I saw your comment and decided to try it out myself.

Turns out we don't need kafka-proxy to get the Sentry components working. Basically I looked into Sentry's source code and the DEFAULT_KAFKA_OPTIONS now accepts a new schema:

{
  "common": { ... }
}

This one allows you to specify the same operations but they actually take those in and forward it to rdlibkafka or whatever it's called. Then at the end of the sentryConfPy you have to set KAFKA_CLUSTER["default"] to DEFAULT_KAFKA_OPTIONS again, then Sentry should be able to get it connected. As for Relay there isn't an easy way so I opened https://github.com/sentry-kubernetes/charts/pull/1514, hope that helps!