search

datalust / seq-forwarder

Local collection and reliable forwarding of log data to Seq
Apache License 2.0
57 stars 15 forks source link

Fix http client usage timeouts and port open length #38

Closed danjohnso closed 5 years ago

danjohnso commented 6 years ago

The default length is what we use for timeout, but not sure if there is a sweetspot for how often the log shipper runs. The other piece of this is using the httpclient as a singleton which looks like a huge change based on how this is architected. I'll start with this and we can discuss

nblumhardt commented 6 years ago

Thanks for the investigation and PR, Dan!

A couple of questions come to mind-

An instance of HttpLogShipper (and thus its HttpClient) should only be created per-API-key. Unless you have thousands of API keys I wouldn't expect port exhaustion from this... Does that match your understanding?

Calls to ingest events into Seq are normally very quick; it'd be really surprising to see ingestion requests taking longer than a few seconds; are the timings as expected on your machine?

Great that we're starting to narrow this down, thanks again!

danjohnso commented 6 years ago

Let me try to time some of the calls tomorrow, make sure it isn't taking longer. We have maybe 20 api keys, so thats odd. What you are saying makes sense from what I remember (not looking at it right now), but I reset the network adapter to force the ports to reset and within a couple of minutes had over 1000 ports in use again. I will try to add some trace logging and test my dev environment to see why so many instances of httpclient are being spun up.

nblumhardt commented 5 years ago

Closing this one as stale, but please let us know if you're hitting any problems around this.