search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.41k stars 1.07k forks source link

One question for Graylog2 v1.0-beta.1 #885

Closed shaman478 closed 9 years ago

shaman478 commented 9 years ago

Hello,

For the new features about v1.0-beat.1:

Message Journaling Every single incoming message is now persisted to disk by default and no longer buffered in memory. The standard config is to hold messages until reaching either 5 GB or Elasticsearch has not been reachable for 12 hours. To be clear, during normal operation the disk usage will be much smaller. This brings both huge performance and stability improvements. It is now very hard to overload your Graylog system and make it crash.

Question 1: What will happen for these messages after "the standard config is to hold messages until reaching either 5 GB or Elasticsearch has not been reachable for 12 hours"? it will be delete from disk or elasticsearch or not?

Question 2: What does the meaning of "This brings both huge performance and stability improvements"? the disk I/O reduced by message incoming low , so that's can be improve the system stability and performance?

B.R Yan da

shaman478 commented 9 years ago

By the way, do you have any plan for the official version delivery?

kroepke commented 9 years ago

Hi!

The journal only grows if the system cannot write messages to elasticsearch fast enough. If that is the case long enough so that the journal reaches its limit, the oldest messages will be removed. They are then lost and will never be written to elasticsearch. Basically make the journal large enough to cover the longest downtime or biggest spike in incoming messages. Elasticsearch is not involved in storing the journal, that is a feature of graylog.

The stability improvements come from the fact that graylog will not buffer messages in memory any more and thus does not suffer from high garbage collection times whenever a source sends many more messages than your system can handle in a short amount of time. The journal also makes sure that graylog or elasticsearch can get slow without losing incoming messages (until the limit i described above happens). Also if graylog should crash for some reason you won't lose all the messages it currently processes in memory.

We will release 1.0 final as soon as we have no more serious bugs to fix and get good feedback from our users. I cannot give you a specific date yet, but we should be getting relatively close.

Best, Kay On Jan 16, 2015 6:39 AM, "shaman478" notifications@github.com wrote:

Hello,

For the new features about v1.0-beat.1:

Message Journaling Every single incoming message is now persisted to disk by default and no longer buffered in memory. The standard config is to hold messages until reaching either 5 GB or Elasticsearch has not been reachable for 12 hours. To be clear, during normal operation the disk usage will be much smaller. This brings both huge performance and stability improvements. It is now very hard to overload your Graylog system and make it crash.

Question 1: What will happen for these messages after "the standard config is to hold messages until reaching either 5 GB or Elasticsearch has not been reachable for 12 hours"? it will be delete from disk or elasticsearch or not?

Question 2: What does the meaning of "This brings both huge performance and stability improvements"? the disk I/O reduced by message incoming low , so that's can be improve the system stability and performance?

B.R Yan da

— Reply to this email directly or view it on GitHub https://github.com/Graylog2/graylog2-server/issues/885.

shaman478 commented 9 years ago

Hello Kay,

The journal features is already available on 0.9x version or only vaild for 1.0 version?

B.R Yan da

joschi commented 9 years ago

The Kafka-based journal has been introduced in Graylog2 1.0.0-beta.1.

For additional questions which are not bug reports or feature requests, I'd like to encourage you to use our mailing list at https://groups.google.com/forum/#!forum/graylog2

kroepke commented 9 years ago

It is new in 1.0. On Jan 16, 2015 10:29 AM, "shaman478" notifications@github.com wrote:

Hello Kay,

The journal features is already available on 0.9x version or only vaild for 1.0 version?

B.R Yan da

— Reply to this email directly or view it on GitHub https://github.com/Graylog2/graylog2-server/issues/885#issuecomment-70228441 .

shaman478 commented 9 years ago

@kroepke

So as my understand:

  1. As 1.0 version,Graylog will storing the message on hard disk replaced by in memory,The message journal feature control the storing size, so we need make sure as bia as the journal parameter value right?
  2. As you know the memory is faster than disk, how to ensure the search performance if we used the journal feature?

B.R Yan da