search

nsqio / go-nsq

The official Go package for NSQ
MIT License
2.59k stars 444 forks source link

Max Open File Publishing Message to NSQd #298

Closed rizalgowandy closed 4 years ago

rizalgowandy commented 4 years ago

So I know that a producer instance is 1:1 with a destination NSQd. Does it have max open file on top of it? Let's say I publish 100 messages at the same time to the same NSQd with a different topic, how many open files am I creating?

Based on my current understanding after reading the code, It seems we can create max int32 producers, which means max int32 open files.

ploxiln commented 4 years ago

A single publisher connection to nsqd can publish multiple messages, and on multiple topics. Typically you would worry more about all the consumers connecting to nsqd from different servers, which will often be the bulk of the connections. nsqd will also have disk-queue files open for every topic and channel, which can also use a lot of file descriptors. Anyway, usually you will run into other performance problems long before you hit 2 billion file descriptors open for a single nsqd process.

ploxiln commented 4 years ago

(And the specific open-file / file-descriptor limit for nsqd is defined by the OS and the service runner you use. It is pretty common to set it to 1 million in the service runner you use, and then not worry about it. nsqd won't work well with 200,000 topics/channels on a single instance anyway.)

rizalgowandy commented 4 years ago

Hi @ploxiln thank you for the explanation :+1: