search

datalust / seqcli

The Seq command-line client. Administer, log, ingest, search, from any OS.
https://datalust.co/seq
Apache License 2.0
143 stars 21 forks source link

Support durable log forwarding in seqcli #318

Open KodrAus opened 7 months ago

KodrAus commented 7 months ago

It would be beneficial to merge Seq Forwarder into seqcli. This would let us share a lot of the log shipping functionality they have in common while enhancing existing seqcli commands related to ingestion with bookmarking from Seq Forwarder. It will simplify how we deploy Seq Forwarder, making its functionality available to seqcli users on Windows or Docker on any platform already supported by seqcli.

The CLI may look something like:

seqcli forwarder run --listen "http://localhost:5341" --storage "/seqcli-logs"

And on Windows, some additional commands for managing installation as a Windows service:

seqcli forwarder install
seqcli forwarder start
seqcli forwarder stop
seqcli forwarder remove

As part of this, we'd like to replace the LMDB-based disk buffer with something bespoke that's easier to manage and doesn't rely on the host to install libraries to support it. The transition from LMDB to the new format will be handled in Seq Forwarder, so existing users can let their old buffers age out before migrating to seqcli.

We also need to overhaul how secrets are managed by seqcli to better support secure deployments on non-Windows platforms.

This work effectively deprecates the Seq Forwarder repository, which will enter maintenance mode.

KodrAus commented 6 months ago

Apologies to anybody currently watching this repository and getting swamped by our barely coherent broken PRs to a feature/forwarder branch 😅

We're currently working towards getting an MVP of this into the tree. Once it's in reasonable shape we'll write up some more details about what's there and how it all works.