search

danieleteti / loggerpro

An modern and pluggable logging framework for Delphi
Apache License 2.0
352 stars 91 forks source link

Performance Issues #48

Closed fastbike closed 2 years ago

fastbike commented 3 years ago

We have an ISAPI application that is quite chatty when we are writing in debug level (i.e. each request will generate about 300 log lines). We have a file appender configured.

We are now receiving about 160k web requests per day and have noticed that after a while the requests are failing with the error message "'Main logs queue is full. Hints: Are there appenders? Are these appenders fast enough considering the log item production?"

I have traced this error message back to TLogWriter.Log in the LoggerPro.pas file. It appears that either the main Log queue is filling up (i.e. log items are being added faster than the log thread can dequeue them), or that the call to FLoggerThread.LogWriterQueue.Enqueue is failing for some other reason.

We have written a simple JMeter test script to reproduce the load. With the global var DefaultLoggerProMainQueueSize set to 10k the error occurs after about 50 seconds. Increasing this queue size to 100k makes the error occur after about 5 minutes.

We really do not want to increase the queue size any more as the idea of 100k unwritten log entries in memory makes us a bit nervous.

Is there any other way we can improve the performance (apart from having separate instances of the ILogWriter) ?

danieleteti commented 3 years ago

https://github.com/danieleteti/delphimvcframework/issues/481

fastbike commented 1 year ago

I'm going to reopen this ticket as we're seeing issues now that our volumes have increased and we are also writing some lessor volume of entries to the Windows Event log.