search

emabee / flexi_logger

A flexible logger for rust programs that can write to stderr, stdout, and/or to log files
Apache License 2.0
315 stars 55 forks source link

feat: Optional feature to de-dup consecutive logs after some leeway #76

Closed jesdazrez closed 3 years ago

jesdazrez commented 3 years ago

Hi there!

We're using flexi_logger with this optional feature to avoid getting a lot of noise from some loops that can log the same message a lot of times.

These changes add the ability to skip dups after some configurable leeway. Some messages are logged to summarize when and how many records were skipped.

I created the dedup example to demonstrate the functionality:

[2021-05-27 11:02:14.412577 -05:00] INFO [dedup] examples\dedup.rs:15: foo
[2021-05-27 11:02:14.413154 -05:00] INFO [dedup] examples\dedup.rs:15: foo
[2021-05-27 11:02:14.413287 -05:00] INFO [dedup] examples\dedup.rs:15: foo
[2021-05-27 11:02:14.413433 -05:00] WARN [flexi_logger] src\flexi_logger.rs:165: last record has been repeated consecutive times, following duplicates will be skipped...
[2021-05-27 11:02:14.413713 -05:00] INFO [flexi_logger] src\flexi_logger.rs:179: last record was skipped 2 times
[2021-05-27 11:02:14.413868 -05:00] INFO [dedup] examples\dedup.rs:15: bar
[2021-05-27 11:02:14.414023 -05:00] INFO [dedup] examples\dedup.rs:15: foo
[2021-05-27 11:02:14.414302 -05:00] INFO [dedup] examples\dedup.rs:15: foo
[2021-05-27 11:02:14.414533 -05:00] INFO [dedup] examples\dedup.rs:15: foo
[2021-05-27 11:02:14.414694 -05:00] WARN [flexi_logger] src\flexi_logger.rs:165: last record has been repeated consecutive times, following duplicates will be skipped...
[2021-05-27 11:02:14.414887 -05:00] INFO [flexi_logger] src\flexi_logger.rs:179: last record was skipped 1 times
[2021-05-27 11:02:14.415021 -05:00] INFO [dedup] examples\dedup.rs:18: the end

Let me know what you think.

jesdazrez commented 3 years ago

Updated with target set as the module_path for the summary logs

jesdazrez commented 3 years ago

Resolved conflicts

jesdazrez commented 3 years ago

Updated examples and benches

emabee commented 3 years ago

Thanks a lot for this PR! I like the idea, and I generalized it such users can now provide any kind of stateful filter. I moved your Deduper code into examples/dedup.rs and modified it such that it now uses the new option. But maybe you want to move the Deduper into a separate crate to allow others to directly use it?