Closed jafri closed 5 years ago
Hi @jafri, interesting. Are you using 1.5.0-beta.1
? And can you add here how are you posting these messages, is it a Promise.all
, sequential, etc?
Thanks.
Its a for loop using 1.5.0-beta.1 + multiple messages at a time.
My newer tests show that messages are delivered properly when timeout is increased to 60 seconds.
I thought idempotent mode handles timeouts and prevents duplicates?
for (let messages of messageBox) {
this.producer.send({
topic: 'messages',
idempotent: true,
messages: [
...messages.map(message => ({ value: JSON.stringify(message) }))
],
})
}
I think this is a bug, the request queue should ignore timeouts when using idempotent
. Can you increase the timeout for now, I will get this fixed on the next version (1.5.0.beta.2
).
Looking at your code now, can you add await
to the send
calls? This way you are generating a lot of concurrent promises
I will look into it, but as the NodeJS app can easily handle the concurrent promises, I do believe there is an issue somewhere with the experimental idempotency guarantee.
So, I ran some tests, and it looks fine. Since you are not awaiting send
you are spawning a huge amount of promises at once (100k) and completely overloading the system. I used the example producer in the repo to test:
const run = async () => {
await producer.connect()
for (let i = 0; i < 100000; i++) {
await Promise.all([sendMessage(), sendMessage()])
}
}
run().catch(e => console.error(`[example/producer] ${e.message}`, e))
sendMessage
generates between 10 to 1000 messages, so I end up with ~1M published messages.
[ { topicName: 'topic-test',
partition: 1,
errorCode: 0,
baseOffset: '161592',
logAppendTime: '-1' },
{ topicName: 'topic-test',
partition: 4,
errorCode: 0,
baseOffset: '159825',
logAppendTime: '-1' },
{ topicName: 'topic-test',
partition: 2,
errorCode: 0,
baseOffset: '162346',
logAppendTime: '-1' },
{ topicName: 'topic-test',
partition: 5,
errorCode: 0,
baseOffset: '161405',
logAppendTime: '-1' },
{ topicName: 'topic-test',
partition: 0,
errorCode: 0,
baseOffset: '161060',
logAppendTime: '-1' },
{ topicName: 'topic-test',
partition: 3,
errorCode: 0,
baseOffset: '161235',
logAppendTime: '-1' } ]
But I agree that even if it overloads the system, it should work. I will check the Java implementation to see if they also ignore the timeouts when using idempotent producers. It shouldn't be a problem in real-world scenarios, but I will take a look anyway. Thanks.
I checked the java client, and the request timeout is used even for idempotent producers. My opinion is the same here; you just generated too many promises on the same tick. We are still looking at this to see if we can make some improvements, but for now, I will close this issue. Feel free to re-open.
Seeing lots of these errors
This is the command
Sending 100K of these messages, and interestingly enough, I end up with anywhere between 110- 120K messages (retried 5 times).