ntent / kafka4net

C# client for Kafka
Apache License 2.0
52 stars 32 forks source link

Producer throws exception when sending a lot of messages #22

Closed avoxm closed 8 years ago

avoxm commented 8 years ago

I am trying to stress test the producer and it seems like it is timing out all the time Here is the code

            var producer = new Producer(seed2Addresses, 
                new ProducerConfiguration(topic));
            producer.ConnectAsync().Wait();

            for (int i = 0; i < 200000; ++i)
            {
                var msg = new Message()
                {
                    Key = BitConverter.GetBytes(1),
                    Value = Encoding.UTF8.GetBytes(String.Format("Test{0}", i))
                };
                producer.Send(msg);
            }
            producer.CloseAsync(TimeSpan.FromSeconds(120)).Wait();

While running this code it throws 2 exceptions :

Timed out while waiting for Send buffers to drain. Canceling and Timed out even after cancelling send loop! This shouldn't happen, there will likely be message loss

I suspect that it is because of CloseAsync, but not sure ... Is this how the producer is supposed to be used in kafka4net ?

vchekan commented 8 years ago

@avoxm I'll take a look soon. Meanwhile you might be willing to trace what Producer is doing under the hood: https://github.com/ntent-ad/kafka4net/wiki/Troubleshooting#tracing

avoxm commented 8 years ago

Thanks @vchekan, will check the traces from my end.

avoxm commented 8 years ago

Apparently the problem was related to connectivity since I was using VPN. It works perfectly fine with "direct", no VPN connection.