search

datalust / seq-forwarder

Local collection and reliable forwarding of log data to Seq
Apache License 2.0
57 stars 16 forks source link

Allow API keys to be forwarded with events #2

Closed nblumhardt closed 7 years ago

nblumhardt commented 8 years ago

Individual applications on a machine may use different API keys to drive their logging level, filtering, and attached properties on ingestion.

If an application uses an API key when connecting to the forwarder, it would be nice to send the events under that API key in their own batch, overriding any API key configured directly in the forwarder.

From: http://docs.getseq.net/v3/discuss/575ac49ea083950e0044871a

EvanMulawski commented 7 years ago

Any update on this? We would like to use one shared instance of Seq Forwarder on each of our hosts to capture and send all events from all apps running on a host to the configured Seq server.

nblumhardt commented 7 years ago

Thanks for the nudge, Evan. We haven't made progress on this one, though I hope that after the coming Seq 4 release we'll be able to focus some energies on the forwarder. I'll update this ticket when things get moving :-)

mason-mcglothlin commented 7 years ago

Any update on the status of this?

nblumhardt commented 7 years ago

@mason-mcglothlin thanks for the nudge. While not strictly a part of the overall Seq release train, I've added this to our 4.2 milestone and will aim to have a solution at or around the time we get Seq 4.2 out the door. Expecting ~September at this stage. Thanks again!

danjohnso commented 7 years ago

Definitely interested in seeing this function. We have ~6 individual applications that all forward logs through this.

nblumhardt commented 7 years ago

Making progress on this one finally 🎉 ... should have a pre-release build out in the next day or so to try.

The overall design is for the forwarder to be transparent as far as API keys are concerned: if the client app provides one, its events will be sent to Seq using that key. If no API key is provided by the client app, we'll fall back to the API key configured in SeqForwarder.json, if there is one. This should make the upgrade path reasonably smooth, as long as apps are connecting to the forwarder with either no key, or a valid key.

The major challenge is that, if the forwarder acts transparently and groups calls to Seq by API key provided by the application, multiple log buffers need to be maintained: calls with one API key may succeed, while others may fail, with the net result being we need to manage a cursor and storage per client API key.

The plan at this stage is just to create a buffer and shipper per API key - manageable for up to 20 or so API keys per instance, I think.

nblumhardt commented 7 years ago

All - there's now an MSI at:

https://ci.appveyor.com/project/datalust/seq-forwarder/build/artifacts

It's built from the dev branch, with this feature merged. If you're able to check it out, any feedback on how the feature has shaped up would be fantastic. Thanks!

danjohnso commented 7 years ago

Working as advertised so far in our dev environment. Will roll it out to prod in our next release cycle.

nblumhardt commented 7 years ago

Brilliant, thanks @danjohnso