Use elasticsearch_data nodes in Logstash output

cloudfoundry-community / logsearch-boshrelease

A BOSH-scalable ELK release

Apache License 2.0

45 stars 49 forks source link

Use elasticsearch_data nodes in Logstash output #141

Closed Infra-Red closed 5 years ago

Infra-Red commented 5 years ago

Hey! This PR changes the default hosts configuration in logstash elasticsearch output. Instead of collocating ElasticSearch and Logstash on the same VM we can point Logstash output plugin to the DNS name of ElasticSearch data nodes. This will decrease the amount of RAM used by Ingestor instances, check the graph below:

Ingestor Memory Usage

I will also create a separate PR which will allow to configure Elasticsearch coordinating nodes in a high load environments.

/cc @pommi @diogomatsubara

axelaris commented 5 years ago

From my point of view - having a local ES drain for Logstash was a great idea, and keeps cluster configuration simpler. To resolve memory issues, we can just increase amount of memory for instance, and maybe change heap calculating formula here. Anyway, I'd prefer to keep this conversation for a couple of weeks to hear from other active Logsearch users.

Infra-Red commented 5 years ago

Hi @axelaris ! I changed the PR and now by default Logstash will forward to local ES host. I've added link support to ingestor_syslog job and now operator can optionally configure forwarding to remove ES host. I've also added ops file for this ingestor-forward-to-elasticsearch-data.yml. Is it works for you?

axelaris commented 5 years ago

Merged. Thank you @Infra-Red!