search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.06k forks source link

Periodic "Notification condition [NO_MASTER] has been fixed." #2740

Closed eroji closed 8 years ago

eroji commented 8 years ago

We are seeing these periodic messages in Graylog server's system messages:

Notification condition [NO_MASTER] has been fixed.

There is no clear cause for this as far as I can tell. Tailing the Graylog server log only shows the following errors, which I believe only has to do with incoming log messages that are failing to be parsed successfully.

2016-08-24 14:07:54,673 ERROR: org.graylog2.inputs.codecs.GelfCodec - Could not parse JSON, first 400 characters:
2016-08-24 14:07:54,673 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message d32d3261-6a3e-11e6-8d23-0050568f436e (journal offset 1417369781) encoded as gelf received from unknown source.
2016-08-24 14:07:54,673 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=d32d3261-6a3e-11e6-8d23-0050568f436e, journalOffset=1417369781, codec=gelf, payloadSize=8, timestamp=2016-08-24T21:07:54.485Z}
2016-08-24 14:07:54,922 ERROR: org.graylog2.indexer.messages.Messages - Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
2016-08-24 14:07:55,926 ERROR: org.graylog2.inputs.codecs.GelfCodec - Could not parse JSON, first 400 characters:
2016-08-24 14:07:55,926 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message d3fda1e4-6a3e-11e6-8d23-0050568f436e (journal offset 1417385047) encoded as gelf received from unknown source.
2016-08-24 14:07:55,926 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=d3fda1e4-6a3e-11e6-8d23-0050568f436e, journalOffset=1417385047, codec=gelf, payloadSize=8, timestamp=2016-08-24T21:07:55.851Z}
2016-08-24 14:07:56,058 ERROR: org.graylog2.inputs.codecs.GelfCodec - Could not parse JSON, first 400 characters:
2016-08-24 14:07:56,058 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message d413259d-6a3e-11e6-8d23-0050568f436e (journal offset 1417385861) encoded as gelf received from unknown source.

Your Environment

dennisoelkers commented 8 years ago

@eroji: This can be the result of your Graylog master node going down (because of being restarted, not registering himself in the MongoDB collection on time because of being overloaded/having connection isssues/etc.) and recovering from that condition. Every time the master is present again, one of the slave nodes notices this and fixes the notification that is supposed to indicate to the user that there is no working master node at this time.

This is an intended behavior of Graylog, therefore I am closing this issue as this tracker is meant for bug reports or feature requests. If you are sure that this is not correct behavior, reopen the issue with more information helping us to understand the issue.