[Meta, 8.5, 8.6] Logstash General Kubernetes Guidelines

[robbavey]

This project is intended to provide users with the best experience for using Logstash in a Kubernetes environment.

In particular we should document and provide examples including any pain points, surprises and places for improvement for the user experience when using Logstash with Kubernetes.

Phase 1 - Exploration Phase

The first phase is expected to include these basic configuration flows:

• [x] @kaisecheng Basic setup Beats/Agent -> Logstash -> Elasticsearch
  • [x] @kaisecheng Add PQ and DLQ using persistent volumes
  • [x] @kaisecheng Point to Elastic Cloud and local instance
  • [x] @kaisecheng Configure with security/TLS
• [ ] Setup logstash monitoring
  • [ ] Elastic cloud
  • [x] @kaisecheng Local cluster
• [x] @kaisecheng Use Logstash as an agent output in Fleet managed mode Follow existing user guide describing the flow, ensuring that . Follow this guide to ensure that it works with a K8S logstash, documenting any areas where it does not
• [x] Add a second pipeline that requires the use of local state
  • [x] @kaisecheng This exercises the use of pipelines.yml, which wasn’t documented, or given as an example in the helm chart
  • [x] @robbavey Sincedb usage will require state storage.
• [x] Add a pipeline that includes a plugin that requires external jar(s), such as the jdbc or jms family of plugins. These require jar(s) to be placed on the classpath, or to be referenced in the pipeline configuration, but still available to the logstash instance.
  • [x] @robbavey Implement use case for jdbc using third party jar

Phase 2 - Documentation

Phase 2 of the Logstash and Kubernetes project is to provide resources for users to help them with get started with running Logstash on Kubernetes, and provide guidance on best practices on how to run and operate Logstash on Kubernetes

These meta issue will contain the tasks needed to complete the phase

• [ ] Create Documentation structure for Logstash Kubernetes documentation

  • [x] Create table of contents/proposed outline of documentation
  • [x] Create documentation skeleton structure https://github.com/elastic/logstash/issues/14574 https://github.com/elastic/logstash/pull/14532

• [ ] Implement documentation topics

  • [x] Overview/Introduction https://github.com/elastic/logstash/issues/14575
  • [x] [inprogress] Getting started guide https://github.com/elastic/logstash/issues/14576
  • [ ] Setting Up
  • [x] [inprogress] Guide to Persistent Storage (PQ/DLQ/state) https://github.com/elastic/logstash/issues/14577
  • [x] Securing Logstash https://github.com/elastic/logstash/issues/14580
  • [ ] Sizing and scaling Logstash in Kubernetes https://github.com/elastic/logstash/issues/14579
  • [ ] Logstash plugins and Kubernetes architecture guide #14578
  • [ ] Administering
  • [x] [inprogress] Stack Monitoring #14572
  • [ ] Upgrading #14582
  • [ ] Logging #14581
  • [x] Troubleshooting https://github.com/elastic/logstash/issues/14573

• [ ] Create Logstash/Kubernetes recipes #14583

[roaksoax]

Logstash on Kubernetes support is driven by Logstash on ECK. As such, I'm closing this issue in favor of https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-logstash.html