OffsetsForTimes giving Local: Unknown partition

[mhowlett]

Description

It has been reported that Consumer.OffsetsForTimes throws an exception "Local: Unknown partition" intermittently for partitions that exist, and that were not recently created.

[ericrameil]

We have the same issue and can reproduce with version 1.5.3. We downgraded on all our clients to version 1.5.2 and the error does not occur. Do you have any plans when this problem will be solved ? Thank you.

[edenhill]

I believe we fixed some issues with leader lookups in librdkafka v1.6.0: https://github.com/edenhill/librdkafka/commit/5514d2de87d41a6283b92fab32b964908dedae02

[ericrameil]

We tested again now and the is it working with Confluent.Kafka version 1.5.3 and librdkafka version 1.5.3, but not with librdkafka 1.6.0. After our testing the problem have to be in librdkafka version 1.6.0.

[edenhill]

Can you reproduce with Debug: "all" set and provide us the logs?

[ericrameil]

Here the log with the method QueryWatermarkOffsets, I am sorry but does not use OffsetsForTimes on the client anymore. But we had the same error some days ago with both methods. In the logfile you can see two times the error and one time it does work without an error. kafkalog3.txt

[edenhill]

There's alot going on in that log and it's hard to make out where you are calling which API. Could you add a printout where you call QueryWatermarkOffsets and for which topic+partitions, and add a second printout with the results / error from that same call?

[ericrameil]

Attached the code and error and debug log, the error occurs after the the assign of the consumer:

log3.txt

` var conf = new ConsumerConfig();

            if (this.Fertigungsauftrag != null && this.Fertigungsauftrag.Maschine != null && this.Fertigungsauftrag.CancelToken != null)
            {
                if (this.Fertigungsauftrag.Maschine.KafkaCluster != null && this.Fertigungsauftrag.Maschine.KafkaCluster.IsValid)
                {
                    conf = new ConsumerConfig                        
                    {
                        GroupId = "Consumer-" + Environment.MachineName,
                        BootstrapServers = this.Fertigungsauftrag.Maschine.KafkaCluster.BootstrapServer,
                        AutoOffsetReset = AutoOffsetReset.Latest,
                        SessionTimeoutMs = 30000,
                        Debug = "all"
                    };
                }
                else
                {
                    Logger.CustomLogger.Error("Fertigungsauftrag oder Maschine null in Kafka Consume");
                    return;
                }
            }

            using (var c = new ConsumerBuilder<string, string>(conf)
                .Build())
            {

                if (this.Fertigungsauftrag != null && this.Fertigungsauftrag.Maschine != null)
                {
                    var topic = Fertigungsauftrag.Maschine.ID.ToString() + "_121";
                    var topicNio = Fertigungsauftrag.Maschine.ID.ToString() + "_122";

                    List<CustomKafkaTopicOffset> list = new List<CustomKafkaTopicOffset>();
                    list.Add(new CustomKafkaTopicOffset(c.QueryWatermarkOffsets(new TopicPartition(topic, 0), new TimeSpan(0, 0, 10)), topic));

                    list.Add(new CustomKafkaTopicOffset(c.QueryWatermarkOffsets(new TopicPartition(topicNio, 0), new TimeSpan(0, 0, 10)), topicNio));

                    List<TopicPartitionOffset> offsets = new List<TopicPartitionOffset>();

                    foreach (var item in list)
                    {
                        //Offset zuweisen
                        if (item.Offset.High > 1)
                            offsets.Add(new TopicPartitionOffset(item.Topic, 0, item.Offset.High - 1));
                        else
                            offsets.Add(new TopicPartitionOffset(item.Topic, 0, 0));
                    }

                    c.Assign(offsets);

`

[ericrameil]

Anything new ?

[qiuwei]

I got similar problems though in a bit more random way. https://github.com/confluentinc/confluent-kafka-python/issues/1043

Rolling back to version 1.5.0 seems to fix the issue.

[SergeSavel]

I have the same issue. Confluent.Kafka 1.6.3, Kafka 2.7.0. Call to QueryWatermarkOffsets() periodically throws an exception "Local: Partition unknown". It happens just after the consumer is created, I don't assign or subscribe the consumer before calling QueryWatermarkOffsets(). It's hard to reproduce this error deliberately, but it happens sometimes.

[edenhill]

This is fixed and will be available in the next release (~3w). As a workaround, call QueryWatermarkOffsets again with the same arguments.

[a-elsheikh]

This is fixed and will be available in the next release (~3w). As a workaround, call QueryWatermarkOffsets again with the same arguments.

Hey there, has a fix for this been released or is this issue actually still open?

[edenhill]

Should be fixed in v1.7.0

[default-work]

Should be fixed in v1.7.0

I dont think so

[kopax-polyconseil]

Hello, we still have this error today :

│ sentry-post-process-forward-transactions-756dc65f9b-lhwtr cimpl.KafkaException: KafkaError{code=_UNKNOWN_PARTITION,val=-190,str="Failed to get watermark offsets: Local: Unknown partition"}                 

Is there a workaround ?