search

kafbat / kafka-ui

Open-Source Web UI for managing Apache Kafka clusters
http://ui.docs.kafbat.io
Apache License 2.0
485 stars 48 forks source link

BE: Expose consumer's config props like`request.timeout.ms` #527

Open cpapad opened 2 weeks ago

cpapad commented 2 weeks ago

Issue submitter TODO list

Is your proposal related to a problem?

When we query with CEL filters or free text large topics we come up with the following error

[Consumer clientId=kafbat-ui-consumer-1724336921133, groupId=null] Failed to close fetcher with a timeout(ms)=30000

To overcome this we need to override the consumer request time out and other values

kafka-ui-6b464dd6c5-fw5cw kafka-ui 2024-08-22 13:29:01,386 INFO  [boundedElastic-1] o.a.k.c.c.ConsumerConfig: ConsumerConfig values:
kafka-ui-6b464dd6c5-fw5cw kafka-ui allow.auto.create.topics = false
kafka-ui-6b464dd6c5-fw5cw kafka-ui auto.commit.interval.ms = 5000
kafka-ui-6b464dd6c5-fw5cw kafka-ui auto.include.jmx.reporter = true
kafka-ui-6b464dd6c5-fw5cw kafka-ui auto.offset.reset = earliest
kafka-ui-6b464dd6c5-fw5cw kafka-ui bootstrap.servers = [kafka-stg-shared-workable-staging.aivencloud.com:24451]
kafka-ui-6b464dd6c5-fw5cw kafka-ui check.crcs = true
kafka-ui-6b464dd6c5-fw5cw kafka-ui client.dns.lookup = use_all_dns_ips
kafka-ui-6b464dd6c5-fw5cw kafka-ui client.id = kafbat-ui-consumer-1724333341385
kafka-ui-6b464dd6c5-fw5cw kafka-ui client.rack =
kafka-ui-6b464dd6c5-fw5cw kafka-ui connections.max.idle.ms = 540000
kafka-ui-6b464dd6c5-fw5cw kafka-ui default.api.timeout.ms = 60000
kafka-ui-6b464dd6c5-fw5cw kafka-ui enable.auto.commit = false
kafka-ui-6b464dd6c5-fw5cw kafka-ui exclude.internal.topics = true
kafka-ui-6b464dd6c5-fw5cw kafka-ui fetch.max.bytes = 52428800
kafka-ui-6b464dd6c5-fw5cw kafka-ui fetch.max.wait.ms = 500
kafka-ui-6b464dd6c5-fw5cw kafka-ui fetch.min.bytes = 1
kafka-ui-6b464dd6c5-fw5cw kafka-ui group.id = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui group.instance.id = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui heartbeat.interval.ms = 3000
kafka-ui-6b464dd6c5-fw5cw kafka-ui interceptor.classes = []
kafka-ui-6b464dd6c5-fw5cw kafka-ui internal.leave.group.on.close = true
kafka-ui-6b464dd6c5-fw5cw kafka-ui internal.throw.on.fetch.stable.offset.unsupported = false
kafka-ui-6b464dd6c5-fw5cw kafka-ui isolation.level = read_uncommitted
kafka-ui-6b464dd6c5-fw5cw kafka-ui key.deserializer = class org.apache.kafka.common.serialization.BytesDeserializer
kafka-ui-6b464dd6c5-fw5cw kafka-ui max.partition.fetch.bytes = 1048576
kafka-ui-6b464dd6c5-fw5cw kafka-ui max.poll.interval.ms = 300000
kafka-ui-6b464dd6c5-fw5cw kafka-ui max.poll.records = 500
kafka-ui-6b464dd6c5-fw5cw kafka-ui metadata.max.age.ms = 300000
kafka-ui-6b464dd6c5-fw5cw kafka-ui metric.reporters = []
kafka-ui-6b464dd6c5-fw5cw kafka-ui metrics.num.samples = 2
kafka-ui-6b464dd6c5-fw5cw kafka-ui metrics.recording.level = INFO
kafka-ui-6b464dd6c5-fw5cw kafka-ui metrics.sample.window.ms = 30000
kafka-ui-6b464dd6c5-fw5cw kafka-ui partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor]
kafka-ui-6b464dd6c5-fw5cw kafka-ui receive.buffer.bytes = 65536
kafka-ui-6b464dd6c5-fw5cw kafka-ui reconnect.backoff.max.ms = 1000
kafka-ui-6b464dd6c5-fw5cw kafka-ui reconnect.backoff.ms = 50
kafka-ui-6b464dd6c5-fw5cw kafka-ui request.timeout.ms = 30000
kafka-ui-6b464dd6c5-fw5cw kafka-ui retry.backoff.ms = 100
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.client.callback.handler.class = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.jaas.config = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.kerberos.kinit.cmd = /usr/bin/kinit
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.kerberos.min.time.before.relogin = 60000
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.kerberos.service.name = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.kerberos.ticket.renew.jitter = 0.05
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.kerberos.ticket.renew.window.factor = 0.8
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.callback.handler.class = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.class = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.connect.timeout.ms = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.read.timeout.ms = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.refresh.buffer.seconds = 300
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.refresh.min.period.seconds = 60
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.refresh.window.factor = 0.8
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.refresh.window.jitter = 0.05
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.retry.backoff.max.ms = 10000
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.login.retry.backoff.ms = 100
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.mechanism = GSSAPI
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.clock.skew.seconds = 30
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.expected.audience = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.expected.issuer = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.jwks.endpoint.url = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.scope.claim.name = scope
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.sub.claim.name = sub
kafka-ui-6b464dd6c5-fw5cw kafka-ui sasl.oauthbearer.token.endpoint.url = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui security.protocol = SSL
kafka-ui-6b464dd6c5-fw5cw kafka-ui security.providers = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui send.buffer.bytes = 131072
kafka-ui-6b464dd6c5-fw5cw kafka-ui session.timeout.ms = 45000
kafka-ui-6b464dd6c5-fw5cw kafka-ui socket.connection.setup.timeout.max.ms = 30000
kafka-ui-6b464dd6c5-fw5cw kafka-ui socket.connection.setup.timeout.ms = 10000
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.cipher.suites = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.endpoint.identification.algorithm = https
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.engine.factory.class = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.key.password = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.keymanager.algorithm = SunX509
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.keystore.certificate.chain = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.keystore.key = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.keystore.location = /etc/kafka/stores/client.keystore.p12
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.keystore.password = [hidden]
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.keystore.type = PKCS12
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.protocol = TLSv1.3
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.provider = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.secure.random.implementation = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.trustmanager.algorithm = PKIX
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.truststore.certificates = null
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.truststore.location = /etc/kafka/stores/client.truststore.jks
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.truststore.password = [hidden]
kafka-ui-6b464dd6c5-fw5cw kafka-ui ssl.truststore.type = JKS
kafka-ui-6b464dd6c5-fw5cw kafka-ui value.deserializer = class org.apache.kafka.common.serialization.BytesDeserializer

While admin configuration options are exposed as far as we are concerned we can not override with configuration or environment variables timeout settings e.t.c. of the consumer.

Describe the feature you're interested in

Values to override request timeout and relevant settings of the consumer used for querying topics to avoid errors and timeouts in large topics.

Describe alternatives you've considered

No response

Version you're running

2956664 v1.0.0

Additional context

No response

github-actions[bot] commented 2 weeks ago

Hi cpapad! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues. Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.