More granular headers

[tocic]

I want to mark my type as safe to copy in that type's header file. But I don't want to include the entire quill and/or fmt in this file. I see there's a file TypeTraitsCopyable.h with the copy_loggable definition I need, but it includes <fmt/ostream.h> and other files I don't want to pollute my header with (I don't use iostreams in my project, for example). Is it possible to restructure files in such a way so that they have only minimal dependencies?

[odygrd]

This is an issue in the library that pulls a lot of headers.

There is some major work is process to solve that in https://github.com/odygrd/quill/tree/experiment see https://github.com/odygrd/quill/blob/experiment/CHANGELOG.md where you can log with minimal includes.

However passing user-defined types like this has been removed and therefore the copy_loggable. Please consider formatting your type to string when passing it to the LOG_ macro for example fmt::to_string(arg) or passing the members of your type as arguments directly one by one

[tocic]

I see, thank you! Is there an approximate release date for 4.0.0?

[odygrd]

Hopefully within the next 3 weeks. It was ready with tests passing. But the removal of user defined types also removed the support for std types requiring them to also formatted on the hot path eg std::vector<int,4>

I didn't like that restriction so i am going through another change that will allow passing std types with value_t as build in types and format them on the hot path.

The problem doing that is all relevant std headers are included when the user isn't using them. To support this with granular headers a lot of functions had to be migrated to templated classes will partially specialisations so that is now extentable when the relevant header is included

To better explain the difference:

Currently quill takes a copy of the type passed using the copy constructor of the type. For types like std::vector the copy will allocate new memory on the hot path then the backend thread will see the copy format it and call the destructor. This destructor includes synchronising back the freed memory to the frontend. That happens for all types except the primitive types and std::string that are copied as pod

With v4.0.0 we only copy trivial types so we never have to call destructors. That is not so simple because now we need to explicitly say how each type will get serialised. For example for std::vector instead of calling the copy constructor on the hot path that would previously copy size, capacity and data* we now iterate the vector and copy the elements one by one directly to our queue buffer. The backend thread creates a local std::vector and populates it by reading the elements from the queue one by one and then that is passed to libfmt for formatting.

I need to add support for doing the above for few more stl types currently i have std::array working with Array.h here

https://github.com/odygrd/quill/tree/experiment/quill/include/quill/std

Now related to user defined types it will be possible for the user to create template specialisations to extend the library to serialise their type to the queue and decode it by extending the library. However as you see from Array.h it includes a bit of work to do for each of your types. Therefore the recommendation will be to format the type on the hot path

[odygrd]

This example shows how to log user defined types with formatting offloaded to backend if you don't wish to convert them to strings in the hot path. You can have a look as a preview if you like

https://github.com/odygrd/quill/blob/experiment/examples/advanced/advanced.cpp

[tocic]

If possible, it's worth adding some helper functions/macros now to reduce the amount of boilerplate code for this use case.

[odygrd]

I do not think it is possible to reduce it with macros, as it has to be specific for each type [example] (https://github.com/odygrd/quill/blob/experiment/examples/advanced/user_quill_codec.h).

But I will see if something can be done

There are generic functions available:

ArgSizeCalculator<decltype(arg)>::calculate(...) Encoder<decltype(arg)>::encode(...) Decoder<decltype(arg)>::decode(...)

You just need to call these functions for each member of the type you want to serialize.

The recommendation is to do this only for types where you cannot afford the conversion to string on the critical path. If you are fine with the conversion, instead of providing all those specializations, you can simply call fmt::to_string(user_defined_type) and format the user-defined type on the frontend, passing it as a string to the logger.

edit: I have now added some helper functions to simplify it further 👍

[odygrd]

Closing for now, this has been done in v4.0.0