Website hangs when log server unavailable using AsyncGelfHttpAppender

[ekirkco]

We had a setup issue with our service and noticed whenever it's not functioning properly the website pushing logs hangs for a very long time on each request. Removing the appender from the list resolves the issue.

[jjchiw]

Hi

Which appender are you using?

So the issue is when the graylog endpoint (udp, http, amqp) "hangs" the appender blocks the website?

Could you elaborate more?

Thanks

[jjchiw]

Sorry went directly to the issue's content and didn't read the subject :(

[jjchiw]

Well I think I have an idea what to tackle and test.

I'll look in the usage of HttpClient.

[ekirkco]

I was assuming it was somehow a threading issue. Are the logs not shipped in a separate thread? Seems like logging should just add to a buffer and go on its merry way.

On Thu, Mar 4, 2021 at 3:47 AM jjchiw notifications@github.com wrote:

Well I think I have an idea what to tackle and test.

I'll look in the usage of HttpClient.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jjchiw/gelf4net/issues/75#issuecomment-790482730, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAROCOFYLDABEBDEUCOGDH3TB5JMTANCNFSM4YR7MI2Q .

[jjchiw]

That's more the job of how log4net manage how the logs are sent... It seems an issue in the starve of threads and timeout of the HttpClient, I'll set a timeout and verify that it only creates one instance, I'll run some tests, there are 2 more issues concerning HttpAppender but didn't know what to test and now I'll try with grayling down and see the threads and connections that are open, during the weekend I'll check it out

On Thu, Mar 4, 2021, 15:23 Evan Kirkconnell notifications@github.com wrote:

I was assuming it was somehow a threading issue. Are the logs not shipped in a separate thread? Seems like logging should just add to a buffer and go on its merry way.

On Thu, Mar 4, 2021 at 3:47 AM jjchiw notifications@github.com wrote:

Well I think I have an idea what to tackle and test.

I'll look in the usage of HttpClient.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jjchiw/gelf4net/issues/75#issuecomment-790482730, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAROCOFYLDABEBDEUCOGDH3TB5JMTANCNFSM4YR7MI2Q

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jjchiw/gelf4net/issues/75#issuecomment-790653388, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAE42D75PDE6BIAG3S7ZN3LTB6JWBANCNFSM4YR7MI2Q .

[jjchiw]

Hi.

Ok, I have a solution based on http://apache-logging.6191.n7.nabble.com/Logger-in-ASP-NET-stops-after-a-few-hours-won-t-restart-until-app-is-cycled-td23330.html.

When there is an exception when connecting to graylog server, it will save the next payloads until there is connection again and will send all of them.

It seems to work fine, I'm testing with https://nbomber.com and the webapp it doesn't hang when I stop and start the input.

So I hope that during the week I'll finish it and let you know.

[ekirkco]

Cool, thanks!

On Mon, Mar 8, 2021 at 7:09 PM jjchiw notifications@github.com wrote:

Hi.

Ok, I have a solution based on http://apache-logging.6191.n7.nabble.com/Logger-in-ASP-NET-stops-after-a-few-hours-won-t-restart-until-app-is-cycled-td23330.html .

When there is an exception when connecting to graylog server, it will save the next payloads until there is connection again and will send all of them.

It seems to work fine, I'm testing with https://nbomber.com and the webapp it doesn't hang when I stop and start the input.

So I hope that during the week I'll finish it and let you know.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jjchiw/gelf4net/issues/75#issuecomment-793229376, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAROCODNQKB664OWOLS4AKTTCVYMFANCNFSM4YR7MI2Q .

[jjchiw]

Hi!

I just pushed the changes if you want to check them out https://github.com/jjchiw/gelf4net/tree/refactor-async-appenders-with-queues-and-timeouts

bufferSize doesn't behave as it used, now it works this way

<!-- Limit of log lines to buffer for async send. Defaults to 1000-->
<!-- If we cannot connect to graylog and the queue reaches the buffersize it will dequeue messages from the queue-->
<bufferSize value="2000" />

And now we have two new properties for the Async~Appenders

  <!-- If we have an exception whe connecting to graylog it will wait and enqueue messages and will start again after milliseconds. Default 5000  -->
<retryAfterMilliseconds value="15000" />
<!-- HttpClient Timeout Limit. Default 2000 only AsyncHttpApender  -->
<httpClientTimeoutMilliseconds value="2000" />

Didn't push the nuget package, because I've to test in .net framework... will try to do it during the week...

Well, If you could try it before I push the nuget it would be great :)

[jjchiw]

Hi!

Just pushed the new packages....

Let me know if it fix the issue

Thanks

[ekirkco]

Hey, sorry about that. I've been heads down on something. Seems to work much better for me though! Thanks! Couple questions on it:

1. Is there any way to know it's failing?
2. How does it handle constant failure? Like, will it just keep growing a buffer and consuming memory or will it start to discard log lines?

On Mon, Mar 15, 2021 at 5:25 AM jjchiw @.***> wrote:

Hi!

Just pushed the new packages....

Let me know if it fix the issue

Thanks

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jjchiw/gelf4net/issues/75#issuecomment-799303961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAROCOHFVTF2RSCJURNUFBDTDXOCFANCNFSM4YR7MI2Q .

[jjchiw]

Great!!!

1. Is there any way to know it's failing?

We are using ErrorHandler from log4net https://stackoverflow.com/a/33261606/181766

2. How does it handle constant failure? Like, will it just keep growing a buffer and consuming memory or will it start to discard log lines?

We are keeping the last <bufferSize value="2000" />

So I'll close the issue :)