Add a multi-chapter deployment reference guide for Logstash docs. This should cover both scaling for throughput and also for achieving high availability. This doc should start with a simple scenario to deploy LS using a file input and ES as sample output. We can then introduce more complex deployment architectures using diagrams. Specifically it should cover:
[x] Simple 1 instance Logstash with file input and ES output, no filters. We should assume this to be a clustered ES setup and use the http protocol to index events
[x] Same setup as above, but we introduce filters -- maybe grok filter. We can then document the use of multiple workers to increase throughput (using -w 8 flag)
[x] Introduce LSF as a light weight shipper which collects logs from edge nodes and ships to centralized LS server.
[x] Assume user has horizontally scaled ES. Talk about following configuration options:
[x] Manual list of nodes in ES output hosts options. We round robin on each request for HA
[x] Introduce hardware load balancer which talks to multiple ES client nodes
[x] Introduce transport client with sniffing option
[x] Scale out Logstash throughput by introducing parallel pipelines (by starting new LS instances), each sharing the workload or processing completely separate configs (file, twitter, etc)
[x] Introduce the use of message brokers to split the pipeline into shippers and indexers phases. List available options like Redis, Kafka, RabbitMQ
[x] High availability Logstash setup using a load balancer in front of the inputs. Also talk about instances going down and how to monitor them (using heartbeat plugin for now)
Add a multi-chapter deployment reference guide for Logstash docs. This should cover both scaling for throughput and also for achieving high availability. This doc should start with a simple scenario to deploy LS using a file input and ES as sample output. We can then introduce more complex deployment architectures using diagrams. Specifically it should cover:
http
protocol to index events-w 8
flag)hosts
options. We round robin on each request for HA