odygrd / quill

Asynchronous Low Latency C++ Logging Library
MIT License
1.14k stars 130 forks source link

Feature request - restore copy_loggable #456

Closed diehard2 closed 1 month ago

diehard2 commented 1 month ago

I've been relying on Quill to move some really heavy weight object serialization outside of the hot path. Specifically stack traces. It looks like in version 4 I have to convert it to a string before I send it to the background thread and to resolve line numbers takes forever (relatively speaking)

It would be great if you could restore this feature, since it was the main reason I switched to quill. Its not the end of the world, I can pin to version 3 indefinitely, but it would be nice to stay current

diehard2 commented 1 month ago

okay, I read the release notes for 4 and I see there's a workaround. I guess I can go that route, but the previous one was really easy to use. If you're taking requests on this one, I would love to see the previous behavior restored.

odygrd commented 1 month ago

I understand that the previous method was convenient. Initially, when I started the project I aimed to find a simple way to avoid explicitly serializing every type, especially each STL type. However, after working with it for a few years, I realized it has significant drawbacks. To achieve optimal latency, a more rigorous approach is necessary.

Using copy constructors to copy objects, as in previous versions, incurs hidden performance costs such as memory allocations on the frontend threads. Additionally, having the backend logging thread call the destructors negatively impacts the frontend critical threads. From version 4 onward, we no longer push any complex types to the lock-free queue - only primitive types. As a result, restoring copy_loggable is not feasible.

For more details on the performance impact, for example with vectors of large strings, have a look here.

Please try to serialize your data as shown in this example: user_quill_codec.h. If you encounter any issues, I am happy to assist.