search

KxSystems / kafka

kdb+ to Apache Kafka adapter, for pub/sub
https://code.kx.com/q/interfaces
Apache License 2.0
50 stars 30 forks source link

Metadata on consumer can present UNKNOWN_TOPIC_OR_PART error #105

Open sshanks-kx opened 8 months ago

sshanks-kx commented 8 months ago

Running .kfk.Metadata[client]; from consumer can present callback msg as follows:

mtype                 topic client partition offset msgtime data                                                                        key      headers                 rcvtime                      
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UNKNOWN_TOPIC_OR_PART test1 0      0         -1001          "Subscribed topic not available: test1: Broker: Unknown topic or partition" `byte$() (`symbol$())!`symbol$() 2024.01.22D15:46:39.829536000

when doing this after doing a subscribe, it can make it look like the subscription had failed (when it hasn't).

Temp removing .kfk.Metadata[client]; after any subscription, the callback no longer fires with a UNKNOWN_TOPIC_OR_PART. It doesn't appear to stop the subscription from occurring, whether its there or not.

Will find out why & change this effect

sshanks-kx commented 8 months ago

Occurs every time .kfk.Metadata[client] called after subscription is made (e.g. call twice, get 2 UNKNOWN_TOPIC_OR_PART msgs). Doesn't occur when called prior to any subscription.

sshanks-kx commented 8 months ago

Created librdkafka issue https://github.com/confluentinc/librdkafka/issues/4589 (can see same occurrence without using kdb+). Current workaround is that if you need to use .kfk.Metadata to see topics/etc, call it prior to subscription.

sshanks-kx commented 6 months ago

Update now on linked issue. May be issue with librdkafka. To try possible fix.

sshanks-kx commented 6 months ago

Tested with the pending librdkafka pull request - bug still appears to be there. Informed librdkafka dev. Current workaround is to do any metadata request prior to subscription.

sshanks-kx commented 6 months ago

bug doesnt occur with v2.0.2 of librdkafka. will await response from librdkafka devs.

sshanks-kx commented 6 months ago

looks like its working again on latest pr change to librdkafka (issue linked above). tested with kdb+. will await future librdkafka release to see if it makes it in.

sshanks-kx commented 2 weeks ago

Fixed in underlying librdkafka 2.4.0