ELK Stack (ElasticSearch, Logstash, Kibana): Maybe too heavy and brittle for our use case?
Graylog: Uses ElasticSearch under-the-hood for searching.
Grafana Loki: "Like prometheus, but for logs" (We already run Prometheus and Grafana on NSIDC hardware for monitoring VMs). Requires special labels on log messages to effectively do its indexing.
Questions
How can we make our logging system resilient to changes in log storage backends? E.g. using Vector (https://github.com/vectordotdev/vector) would allow us to log to Vector and broadcast those logs to multiple back-ends, or switch backends in-flight.
Can our tool of choice ingest from our existing text log files to populate a history?
Can it ingest from live text files being written to by existing apps not configured for network logging? (e.g. ELK Stack's "filebeat" forwarder)
What protocol(s) is/are used to pass logs to the service?
Discussion: https://nsidc.slack.com/archives/C4UCJ1NAF/p1666721015728609
Some things to look at:
Questions
References/Links