lstmtraining mutex lock failed

[stweil]

Sometimes the training process lstmtraining raises an exception when the training is finished:

Finished! Selected model with minimal training error rate (BCER) = 2.849
libc++abi: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

Maybe that happens when the training finishes while the validation thread is still running.

[stweil]

In the same context another error can also occur (I got it yesterday while examining the above issue):

Finished! Selected model with minimal training error rate (BCER) = 12.951
num_docs > 0:Error:Assert failed:in file ../../../src/ccstruct/imagedata.cpp, line 697

This is an error which was already reported in issue #757.

[amitdo]

A general note: maybe we can adopt some of the suggestions from the C++ Core Guidelines: https://github.com/isocpp/CppCoreGuidelines/blob/master/CppCoreGuidelines.md#cp-concurrency-and-parallelism