Help Request: Summarizing repeating messages

[erikbgithub]

I currently have the typical scenario that I receive the same log message from the same logger, without any changes beside the timestamp, like 10 times. Then something else happens. Then again a few times the same message. Would it be possible to summarize these messages? E.g. from:

[2014-10-30 15:30] DEBUG: loggerA: do something
[2014-10-30 15:30] DEBUG: loggerA: do something
[2014-10-30 15:30] DEBUG: loggerA: do something
[2014-10-30 15:30] DEBUG: loggerA: do something

to:

[2014-10-30 15:30] DEBUG: loggerA: do something (4x)

It's a little like the FingersCrossedHandler. I'm not sure if this is already possible without writing a new Handler.

[vmalloc]

Have you looked at the dedup handler?

— Rotem

On Thu, Oct 30, 2014 at 5:39 PM, Erik Bernoth notifications@github.com wrote:

I currently have the typical scenario that I receive the same log message from the same logger, without any changes beside the timestamp, like 10 times. Then something else happens. Then again a few times the same message. Would it be possible to summarize these messages? E.g. from:

[2014-10-30 15:30] DEBUG: loggerA: do something
[2014-10-30 15:30] DEBUG: loggerA: do something
[2014-10-30 15:30] DEBUG: loggerA: do something
[2014-10-30 15:30] DEBUG: loggerA: do something

to:

[2014-10-30 15:30] DEBUG: loggerA: do something (4x)

It's a little like the FingersCrossedHandler. I'm not sure if this is already possible without writing a new Handler.

Reply to this email directly or view it on GitHub: https://github.com/mitsuhiko/logbook/issues/128

[erikbgithub]

Yes, it looks exactly like what I was looking for! Thanks!

[vmalloc]

@alonho another satisfied customer :-)

[erikbgithub]

Sorry, I was too optimistic too soon. The following example doesn't print anything.

from logbook import *
from logbook.more import *
NestedSetup([
    DedupHandler()
]).push_application()

error('foo')
error('foo')
error('foo')
error('foo')
error('foo')

btw setting bubble is not possible

[vmalloc]

Well, the current implementation relies on calling flush() explicitly to write out the stored records... Not sure this is exactly what you need, but if not please elaborate more on you scenario

[erikbgithub]

You mean, I'll have to call flush when I think that I'm ready to print something? I thought it would do something like this:

1. get new message
2. buffer message because buffer is empty
3. get new message
4. increase counter if buffer is identical to new message
5. print buffer+counter if new message is different from buffer
6. store new message in buffer because buffer is empty

If I need to call flush manually the "meat" of this task is still left to the user :-(

[vmalloc]

Well, that's ok, but there are some fine details. For example, what happens if no records appear in a very long time? This could mean that the program is idle, but the last log never shows up because it is buffere...

[erikbgithub]

If no records appear in a long time it could iddle like any other handler would. And if you buffer the results you would also make sure in __del__ that the last buffered message is dealt with before the object is gone, anyway. So you wouldn't lose the last message, right?

In any case those are implementation details a framework like logbook should deal with for the user.

[vmalloc]

Yes of course. I was just pointing out some reasons why it wasn't implemented yet -- it's not trivial...