search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.24k stars 1.05k forks source link

Cluster Events Collection kills MongoDB Server with high(er) amount of Graylog Server #18785

Open HenryTheSir opened 3 months ago

HenryTheSir commented 3 months ago

Expected Behavior

Scalable Graylog Servers without a bottleneck in MongoDB CPU

Current Behavior

Every Graylogserver in the cluster queries the cluster_event collection every second for entrys NOT containing his own node id. This leads to a super high cpu usage on mongodb side.

Possible Solution

Manually Clear cluster_event collection: db.cluster_events.deleteMany({}) { "acknowledged" : true, "deletedCount" : 9228 }

Arround 11 our deployment changed many graylog entities by rest api, arround 15 we cleared the cluster_events collection (MongoDB Server has 8 Cores) grafik

Steps to Reproduce (for bugs)

  1. Create a high amount of changes via webinterface or a high amount of ended sessions
  2. observe thousands of entrys in cluster_events
  3. watch mongodb cpu die

Context

Your Environment

bernd commented 3 months ago

Refs https://github.com/Graylog2/graylog2-server/issues/17411