search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.44k stars 1.07k forks source link

Programatically promote masters #3239

Open florianpopp opened 7 years ago

florianpopp commented 7 years ago

From @lennartkoopmann on September 17, 2013 13:27

Show a prominent warning when multiple graylog2-server master nodes are in the cluster.

Copied from original issue: Graylog2/graylog2-web-interface#203

florianpopp commented 7 years ago

From @kroepke on September 17, 2013 18:37

We should not only warn but also resolve this in a future version (maybe not for 0.20) On Sep 17, 2013 3:27 PM, "Lennart Koopmann" notifications@github.com wrote:

Show a prominent warning when multiple graylog2-server master nodes are in the cluster.

— Reply to this email directly or view it on GitHubhttps://github.com/Graylog2/graylog2-web-interface/issues/203 .

florianpopp commented 7 years ago

From @lennartkoopmann on September 17, 2013 20:37

Actually, the current v0.20.0 branch is already doing that by starting as a non-master node if it detects another master.

I was a bit unclear in the description: We should warn that multiple masters are configured even though they are not actually running as master nodes.

florianpopp commented 7 years ago

From @kroepke on September 17, 2013 20:54

OK :) On Sep 17, 2013 10:38 PM, "Lennart Koopmann" notifications@github.com wrote:

Actually, the current v0.20.0 branch is already doing that by starting as a non-master node if it detects another master.

I was a bit unclear in the description: We should warn that multiple masters are configured even though they are not actually running as master nodes.

— Reply to this email directly or view it on GitHubhttps://github.com/Graylog2/graylog2-web-interface/issues/203#issuecomment-24620266 .

florianpopp commented 7 years ago

From @dennisoelkers on September 17, 2013 21:9

I think there might be scenarios where multiple masters make sense, for example in hot standby/load balancing scenarios. Is it reasonable to make the warning configurable? Or define the mode of operation (Multi master vs master slave) which would implicitly define the necessity of a warning? Are there periodic master reelections, especially when the node set changes? If yes, maybe some kind of advertisement skewing needs to be implemented?

Kr, D.

On Sep 17, 2013, at 10:54 PM, Kay Roepke notifications@github.com wrote:

OK :) On Sep 17, 2013 10:38 PM, "Lennart Koopmann" notifications@github.com wrote:

Actually, the current v0.20.0 branch is already doing that by starting as a non-master node if it detects another master.

I was a bit unclear in the description: We should warn that multiple masters are configured even though they are not actually running as master nodes.

— Reply to this email directly or view it on GitHubhttps://github.com/Graylog2/graylog2-web-interface/issues/203#issuecomment-24620266 .

— Reply to this email directly or view it on GitHub.

florianpopp commented 7 years ago

From @lennartkoopmann on September 17, 2013 21:21

Good point, Dennis.

Another thing to keep in mind: An engineer at one of our customers mentioned today that the term "Master" is confusing, as it actually is more a "Maintenance Worker".

We could think about promoting nodes to master (or whatever we call it) programatically. This gives us:

On the long term we might want proper master election though. ):

florianpopp commented 7 years ago

From @lennartkoopmann on September 25, 2013 18:5

Changed this issue title to "Programatically promote masters", scheduled to 0.20.1 (aka "after the next release") and re-implement the master warning in this issue for now: #227

rswestmoreland commented 7 years ago

Where is this at? All the current documentation still says there should only be one master, so is this no longer being considered?

I have an idea to address it. If you let multiple nodes enable master status, you can set a master priority 1-10. Any time the same priority is matched across multiple masters, then it will be randomly selected and be a master based on master ttl. That way they can temporarily take turns until a higher priority master comes back online.

bernd commented 7 years ago

@Antisource This will be on our roadmap, I just can't say yet which release it will be done.

Thanks for your suggestion! :smiley:

The first step will be to manually promote a node to leader status via API and the web interface. The main problem here is not the implementation of the API and UI, but that all internal services than only run on a master node need to be re-configurable during runtime. This is currently not the case since changing the leader status requires a node restart. Changing this requires some careful internal refactorings. That's why we first want to implement the manual status change and once that's done and working reliable we tackle the automatic leader election.

radykal-com commented 6 years ago

Hey @bernd how is the automatic leader election going? Is there any draft or working group of it? Have you decided which consensus algorithm will be implemented (Paxos? Raft?) This feature would be very nice when running Graylog in cloud providers, with random number of nodes that may scale in or scale out in autoscaling groups

joschi commented 6 years ago

@radykal-com There's currently no release scheduled for this feature and it's at least not planned to be part of Graylog 3.0.x.

drewmiranda-gl commented 1 year ago

imo this is pretty important, especially in an enterprise setting. any thoughts given to this recently? What happens if the sole leader is offline or unavailable?