search

SOHU-Co / kafka-node

Node.js client for Apache Kafka 0.8 and later.
MIT License
2.66k stars 627 forks source link

HighLevelProducer with KeyedPartitioner fails on first send #354

Open itamarwe opened 8 years ago

itamarwe commented 8 years ago

When using KeyedParitioner with the HighLevelProducer the first send fails with

BrokerNotAvailableError: Could not find the leader

Consecutive sends work perfectly.

eugeneware commented 8 years ago

Yeah. I'm getting the same error no matter what I do.

eugeneware commented 8 years ago

The random partitioner and default parttitioners work. random and keyed both fail with the BrokerNotAvailableError.

itamarwe commented 8 years ago

Only for the first message, though. (or until the topic metadata is updated?)

itamarwe commented 8 years ago

I've created a pull-request to fix it: https://github.com/SOHU-Co/kafka-node/pull/378

As suspected, the metadata isn't updated when the producer is initialized, so the partitions array is empty, which causes the keyed partitioner to return NaN as the chosen partitions, which fails the producer.

griffinmichl commented 8 years ago

Yup, same error. I have also found the documentation for working with keyed partitions very poor (probably because they don't work). I think this is something that could be improved upon as well.

askhogan commented 8 years ago

I just ended up rolling my own and manually setting partition number in payloads.

bfmnorman commented 8 years ago

I ran into this problem today, and I was able to get around it by calling client.loadMetadataForTopics([topic]) and then calling client.updateMetadatas(metadata) in the callback function. After calling that, I called back my own callback saying that the producer is ready to use.

I did all this with several promises, so it might be more difficult to do with just using callbacks. Basically, I didn't resolve the "producer" promise until after updating the metadata for the topic on the client.

badeball commented 5 years ago

I've attemped to create a minimal example to reproduce this issue in this gist. I say attempted, because there's no error thrown with v4.0.1. I wasn't able to install v0.3.2, which seems to be what was available back then (and I'm not super keen on degrading my Arch system back to dinosaur). Is anyone able to use the example to confirm that this used to be an issue and isn't anymore?

alex3165 commented 4 years ago

Hey, I opened a PR to fix this issue: https://github.com/SOHU-Co/kafka-node/pull/1360