Support for asynchronous logging

[DarrenCook]

(I think this is a feature request, but if it already exists, it is a request for a usage example. Thanks!)

There is mention, in 2013, that version 9.0 will support asynchronous logging, but I cannot see anything after that. I was hoping there would be a thread-safe queue, behind the scenes, and a dedicated thread that takes items from the queue and writes them to the disk file. I.e. adding a string to a queue is quick, so doesn't lock a thread for long, but appending to a disk file can be relatively slow.

[abumq]

Several attempts have been made but nothing very solid so far For those wondering about "mention in 2013", it's here

leaving it open for suggestions (and/or comments)

[abumq]

Aiming to have an experimental - which will be enabled by _ELPP_EXPERIMENTAL_ASYNC - in early stages at the moment - a lot of issues

[abumq]

Hi Darren,

thanks for the email but I have already implemented experimental async logging and would love if you could give it a test on your PC - it's in really early stage at the moment and I need to ensure I test across all the platforms to ensure availability of all the headers (and handle different cases) - it currently uses POSIX thread model but obviously it will use native threading model by corresponding platform.

I had it tested on mac and it's not a big success there, i have tested in linux (using Intel C++ Compiler) and it's going good.

In order for you to test it, get a copy of easylogging++.h header from develop branch and compile any program (may be an experimental program in https://github.com/easylogging/easyloggingpp/blob/develop/src/experimental/async/prog.cpp) and define macro _ELPP_EXPERIMENTAL_ASYNC to enable asynchronous logging (use build.sh)

let me know how you go.

Thanks

[abumq]

NOTE: This issue is not to be closed until asynchronous logging is stable (i.e, out of experimental mode)

[abumq]

Dead lock in multithreaded app (ran pthread.cpp from samples/STL)

g++ pthread.cpp -o bin/pthread.cpp.bin  -D_ELPP_DEBUG_ERRORS -D_ELPP_THREAD_SAFE -D_ELPP_STL_LOGGING -D_ELPP_LOG_UNORDERED_SET -D_ELPP_LOG_UNORDERED_MAP -D_ELPP_STACKTRACE_ON_CRASH -D_ELPP_LOGGING_FLAGS_FROM_ARG -std=c++11 -pthread -Wall -Wextra -pedantic -pedantic-errors -Werror -Wfatal-errors -D_ELPP_EXPERIMENTAL_ASYNC 
./bin/pthread.cpp.bin

[DarrenCook]

Can you describe/show how you've implemented it? (The source code is one huge file, littered with conditional compilation lines, so is hard to find things in.)

Just the high-level description is enough (e.g. std::queue protected by a mutex and std::lock_guard for pushing to the queue, then calling condition_variable.notify(). Then a single dedicated thread, using std::unique_lock to find and remove strings from the queue, and writing them out to file.)

It is so easy to get multi-threading wrong that I try to copy patterns/code from threading experts.

[abumq]

Above comment was just reminder for myself - well you are right - this is high level on how it works - right now it's using single queue (not good i know), but since it is in experimental mode so i may keep changing the strategy as i go - but for now yes you are right in terms of how it works but it's just another Async callback that I added instead of changing whole way of how logging work I had it implemented so it's extendable. (if you interested here is line where it happens)

Any way the strategy that i may change it to is every logger may (or may not have - depending on file it's pushing to) it's own dispatch worker. This will fasten up the process since i have seen great improvement with async logging yet clean bit (cleaning queue when application wants to close) takes longer than expected because one thread is busy in cleaning (and performing I/O) massive logs - checkout src/experimental/async/prog.cpp for example - it says "finished" sooner than synchronous logging but cleaning takes longer than expected. another thing is starting worker waits for 5 sec before it start work because of Storage initialization (construction) - need to look at all these conditions and improve code readability; So need to improve this - can't live with it in stable releases. Also dead lock will be looked at when i get chance to work on the project as i normally do it more often over weekends (because of full time work)

I try never to copy code but to understand it and then write my own version so that i understand it inside out :)

[baiwfg2]

@abumusamq Hi. I'm wondering how the async logging functionality going.