nsqio / go-nsq

The official Go package for NSQ
MIT License
2.59k stars 444 forks source link

Load balance nsqd for producing messages #290

Closed EthanHemo closed 4 years ago

EthanHemo commented 4 years ago

Hi,

I'm currently use 3 nsqd servers in my environment and couple of applications write those nsqd.
While i consume the messages with nslookup service, can i write to all 3 nsqd with a load balancer? will this topology be correct?

And can i use the nsqlookup to retrieve the nsqd server for the producer and be updated when a producer is missing or added?

Thanks, Ethan.

jehiah commented 4 years ago

@EthanHemo nsq doesn't dictate any particular load balancing approach for producing message. I like to use a pattern where I colocate nsqd with the application producing messages so they both scale horizontally, but you can use a TCP load balancer (or HTTP load balancer if you publish messages with the HTTP api), etc. You could also use DNS based load balancing (i.e. a DNS entry with multiple A records).

nsqlookupd is designed for consumers to discover an already active topology of messages on nsqd, so using it for publishing presents a cold-start problem; If you want to use it for publishing you will likely need to manually use the nsqd API to create topics on a nsqd node for it to register in nsqlookupd.

hope that helps!