I'm leaving Github. The main official location for this project is now: https://codeberg.org/RafaGago/mini-async-log
A performant asynchronous data logger with acceptable feature-bloat.
This started with the intention to just develop an asynchronous logger that could be used from different dynamically loaded libraries on the same binary without doing link-time hacks like being forced to link static, hiding symbols and some other "niceties".
Then at some point way after the requirements were met I just improved it for fun.
To be:
(*)An on-stack ostream adapter is available as a last resort, but its use is more verbose and has more overhead than the format literals.
see this example that more or less showcases all available features.
It just borrows ideas from many of the loggers out there.
As an asynchronous logger its main objetive is to be as fast and to have as low latency as possible for the caller thread.
When the user is to write a log message, the producer task is to serialize the data to a memory chunk which then the logging backend (consumer) can decode, format and write. No expensive operations occur on the consumer, and if they do it's when using secondary features.
The format string is required to be a literal (compile time constant), so when encoding something like the entry below...
log_error ("File:a.cpp line:8 This is a string that just displays the next number {}, int32_val);
...the memory requirements are just a pointer to the format string and a deep copy of the integer value. The job of the caller is just to serialize some bytes to a memory chunk and to insert the chunk into a queue.
The queue is a mix of two famous lockfree queues of Dmitry Vyukov (kudos to this genious) for this particular MPSC case. The queue is a blend of a fixed capacity and fixed element size array based preallocated queue and an intrusive node based dynamically allocated queue. The resulting queue is still linearizable.
The format string is type-safe and validated at compile time for compilers that support "constexpr" and "variadic template parameters". Otherwise the errors are caught at run time on the logged output (Visual Studio 2010 mostly).
There are other features: you can block the caller thread until some message has been dequeued by the logger thread, to do C++ stream formatting on the caller thread, etc.
The library can rotate log files.
Using the current C++11 standard files can just be created, modified and deleted. There is no way to list a directory, so the user is required to pass the list of files generated by previous runs.at startup time.
There is an example here.
The library isn't a singleton, so the user should provide a reference to the logger instance on each call.
There are two methods to pass the instance to the logging macros, one is to provide it explicitly and the other one is by providing it on a global function.
If no instance is provided, the global function "get_mal_logger_instance()" will be called without being namespace qualified, so you can use Koenig lookup/ADL. This happens when the user calls the macros with no explicit instance suffix, as e.g. "log_error(fmt string, ...)".
To provide the instance explictly the macros with the "_i" suffix need to be called, e.g. "log_error_i(instance, fmt_string, ...)"
The name of the function can be changed at compile time, by defining MAL_GET_LOGGER_INSTANCE_FUNCNAME.
The worker blocks on its destructor until its work queue is empty when normally exiting a program.
When a signal is caught you can call the frontend function on termination in your signal handler. This will flush the logger queue and early abort any synchronous calls.
As of now, every log call returns a boolean to indicate success.
The only possible failures are either to be unable to allocate memory for a log entry or an asynchronous call that was interrupted by "on_termination". A filtered out call returns true".
The logging functions never throw.
Those that are self-explanatory won't be explained.
You can compile the files in the "src" folder and make a library or just compile everything under /src in your project.
Otherwise you can use cmake.
On Linux there are Legacy GNU makefiles in the "/build/linux" folder too. They respect the GNU makefile conventions. "DESTDIR", "prefix", "includedir" and "libdir" can be used. These are mainly left there because the examples were not ported to CMake.
REMEMBER (for legacy users that use boost): If the library is compiled with e.g. the "MAL_USE_BOOST_THREAD" and "MAL_USE_BOOST_CHRONO" preprocessor variables the client code must define them too. TODO: config.h.in
There is a Visual Studio 2010 Solution in the "/build/windows" folder, but you need to do a step before opening:
If you don't need the Boost libraries you should run the "build\windows\mal-log\props\from_empty.bat" script. If you need them you should run the "build\windows\mal-log\props\from_non_empty.bat" script.
If you don't need the Boost libraries you can open and compile the solution, otherwise you need to edit (with a text editor) the newly generated file ""build\windows\mal-log\props\mal_dependencies.props" before and to update the paths in the file. You can do this through the Visual Studio Property Manager too.
It used to be some benchmark code here but the results were reported as being off compared with what some users where getting (out of date versions?).
The tests also had the problem that they were assuming a normal distribution for the latencies.
It was not very nice to add a lot of submodules to this project just for building the benchmarks.
Because all of these reasons I have created a separate repository with some benchmark code. It uses CMake to compile everything and build a static linked executable, so it's very easy to build and run.
If you are writing a logger and want to add it or have some suggestions about how to improve the bencmark code Pull Requests are welcome.
Here is the benchmark project..
Written with StackEdit.