search

nsqio / nsq

A realtime distributed messaging platform
https://nsq.io
MIT License
24.99k stars 2.9k forks source link

nsqlookupd/nsqd: TLS on intra-cluster communication #876

Open ninjacato opened 7 years ago

ninjacato commented 7 years ago

According to this email conversation NSQ assumes that connections between nsqd and nsqlookupd is on a trusted network. Hence, TLS communication is only possible between clients and nsqd. I'm looking at use cases where nsqd and nsqlookupd would be located on geo-distributed systems where there isn't necessarily any "network security guarantees".

Proposal: Would you be interested in a pull request adding an option for TLS-wrapped communication between nslookupqd and nsqd?

ploxiln commented 7 years ago

An interesting complication is that nsqd connects to nsqlookupd with a binary nsq-specific protocol over TCP, and also makes some http requests to nsqlookupd on the http port. This has been a "would be nice to clean up but no one has cared enough yet" sort of thing.

dentm42 commented 7 years ago

One method I've contemplated to resolve this is using autossh and ssh port forwarding. I use this successfully in other scenarios to set up a pseudo-vpn with low overhead and fairly easy administration. Anybody tried with NSQ (I'm just starting an implementation, will update with any gotchas I run across.)

funnylookinhat commented 5 years ago

Bumping this request - would be incredibly useful. Having TLS to handle outside communication is nice, but there are a bunch of scenarios where this would be necessary to actually have a secure deployment.

CodingMinds commented 2 months ago

I might be able to invest some time to implement the necessary changes to support TLS for communication between nsqlookupd and nsqd.

But what's the desired way to do it? Do we need to cleanup the mixed HTTP/binary communication mentioned by @ploxiln beforehand or would it be sufficient to just add the TLS support for the existing connections as a first goal? Should the existing V1 protocol be extended to support the TLS handshake or is a new V2 preferred (I'd assume so)?

mreiferson commented 1 month ago

Yes, we'd have to deal with the HTTP requests somehow. For example today the code assumes http:// when nsqd fetches all the channels for a topic (in order to "pre-create" them).

Updating the native protocol to support the necessary commands and then enabling a TLS handshake seems like the best path forward.