ah- / rdkafka-dotnet

C# Apache Kafka client
Other
242 stars 73 forks source link

Keep reference to PartitionerCallback after topic has been GC #93

Closed treziac closed 7 years ago

treziac commented 7 years ago

In certain case, we call partitioner callback after topic has been disposed / GCollected, and this produce a CallbackOnCollectedDelegate I'm not really happy with this workaround as we should be able to detect when releasing the delegate reference is safe. Not sure blocking collection is useful but never know when people could use thread for whatever reason, so I ended up with this. Feel free to propose change

This is the code I used to generate the exception and validate the fix. This exception, given by microsoft, is random, may or may not occur during runtime and may depend of hardware, so this may not be reproductible everywhere

var topicConfig = new TopicConfig
{
    CustomPartitioner = (top, key, cnt) => 0
};

using (var kafkaProducer = new Producer("zk:9092"))
{
    for (int j = 0; j< 100; j++)
    {
        //This is to create lot of topic to help generate the error
        using (Topic topic = kafkaProducer.Topic("testPartitioner", topicConfig))
        {
            for (int i = 0; i < 10000; i++)
            {
                topic.Produce(null);
            }
        }
    }
}

Also updated referenced version, if you want to keep with old one feel free to tell

ah- commented 7 years ago

Interesting, thanks for debugging this! This is definitely an issue.

I share your feeling that this is a bit hacky, but can't come up with something better on the spot. It looks like the lifetime of the delegate must be greater than the producer handle, so it's roughly in the right place.