Y24-079 Fix logstash service on monit servers

[yoldas]

Describe the Housekeeping As PSD, we would like to configure log stash settings to increase service availability for UAT, Training, and Prod environments.

• Increase Disk space by changing VM flavour on OpenStack for the monitor server.
• Configure percentage of disk space after which logstash goes into read-only mode.
• Configure the heap size of the logstash and elasticsearch java services.

Who are the primary contacts for this story Dasun, AY6

Blocking issues N/A

Additional context

• The issue was found because Kibana did not show any new log records since mid-April 2024 for UAT and Production. The reason for this was that Elasticsearch indexes were somehow made read-only (we are not aware of what prompted this to occur). Logstash was trying to index log messages into read-only indexes and eventually failed.
• We executed the following command on monit hosts to get the logstash to get elasticsearch indices out of read-only mode.

  curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

• We also needed to restart the logstash service on Production monit host. This was because logstash has run out of memory.

sudo systemctl restart logstash

• Ref: https://stackoverflow.com/questions/50609417/elasticsearch-error-cluster-block-exception-forbidden-12-index-read-only-all

[dasunpubudumal]

This story can be made into a more "general" story in terms of investigating and fixing - if there are issues with - our general logging process. The investigation may include analysing the following aspects of logging:

• [ ] The overall architecture of log aggregation: This may include the log aggregation process that we are currently following. The current process uses the ELK stack (which is the aggregator) and Filebeat (which is the log shipper) to aggregate and query logs.
• [ ] Archival/cycling of "stale" logs: This may include analysing how we manage stale logs i.e., past logs that are past a certain time period (e.g., several months). Investigating how the disk space vary with respect to our log retention configuration falls within this point.
• [ ] Current configuration of the ELK Stack: This may include configurations such as Elasticsearch service heap-size allocation. This needs to be investigated to properly allocate proper heap ranges (-Xms and -Xmx).

Investigate the points above and identify the points that require further looking into.