search

nats-io / jetstream

JetStream Utilities
Apache License 2.0
455 stars 44 forks source link

New jetstream clustering -what about super clusters with gateways? #482

Closed dryumkin closed 3 years ago

dryumkin commented 3 years ago

Hi, I've just read about (have not played with yet) the RAFT-based clusters in Jetstream. It seems a bit similar to NATS cluster in that each cluster member sees and can communicate with all other members. In NATS, we also have super-clusters with gateway nodes and the group-queue logic is such that a server will always try to serve local queue subscribers first and only fail-over when a local queue subscriber is not found. The server will pick the cluster with the lowest RTT. This is extremely helpful when you have multiple locations and run a separate cluster in each location and have those clusters "combined" in a super-cluster. How can I achieve a similar goal with Jetsream?

Thanks, Dmitry

ripienaar commented 3 years ago

The data lives in one specific cluster only so we can’t exactly service it from your nearest cluster

how sever when creating streams you have some options. By default the stream is placed in the cluster you connect to so it’s near. You can specify a specific cluster for placing a stream with —cluster.

A final option is to create arbitrary tags for your node - ssd, east, west, etc - and when placing a stream you can say it has to be in nodes with ssd, or east or whatever.

It’s not quite as flexible as the geo failover queue subs - for now data has to be in tbe same cluster - but it’s helpful already.

We also just landed data replication so you can create a read only mirror of another stream in a nearby cluster.

dryumkin commented 3 years ago

Thanks for your detailed response. You said: "for now data has to be in the same cluster - but it’s helpful already". Are you planning to address this in the future releases or is it a fundamental property of the design and will stay the way it is with this respect?

ripienaar commented 3 years ago

I suspect its fairly fundemantal in keeping the replication performant, when you publish a message we have to get a quorum of nodes to have comitted it to disk - so we want that to be fast. If they were US/EU spread, it would really suck.

I did some Active-Active replication experiments - has a bunch of constraints - but 2 streams can mirror each other in a circle and this works, lots of limitations and BUTs in there though.

We'll after GA keep itterating quite quickly on a lot of these things based on feedback, customer demand and just wanting to do the right thing. But where we are today we're getting what we have stable and shipping in a couple of weeks with that.

dryumkin commented 3 years ago

Thanks!

From: R.I.Pienaar [mailto:notifications@github.com] Sent: February-26-21 9:34 AM To: nats-io/jetstream jetstream@noreply.github.com Cc: RYUMKIN Dmitry dmitry.ryumkin@thalesgroup.com; Author author@noreply.github.com Subject: Re: [nats-io/jetstream] New jetstream clustering -what about super clusters with gateways? (#482)

I suspect its fairly fundemantal in keeping the replication performant, when you publish a message we have to get a quorum of nodes to have comitted it to disk - so we want that to be fast. If they were US/EU spread, it would really suck.

I did some Active-Active replication experiments - has a bunch of constraints - but 2 streams can mirror each other in a circle and this works, lots of limitations and BUTs in there though.

We'll after GA keep itterating quite quickly on a lot of these things based on feedback, customer demand and just wanting to do the right thing. But where we are today we're getting what we have stable and shipping in a couple of weeks with that.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/nats-io/jetstream/issues/482#issuecomment-786683190, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AS727KXZ4SU2HBFEOJG7YFLTA6WM5ANCNFSM4YHMQXHA.