search

damellis / ESP

The Example-based Sensor Predictions (ESP) system applies machine learning to real-time sensor data.
BSD 3-Clause "New" or "Revised" License
224 stars 52 forks source link

Pause pipeline when training starts, resume when complete. #301

Open damellis opened 8 years ago

damellis commented 8 years ago

Audio data might be accumulating in a separate thread during training.

damellis commented 8 years ago

Or do training in the background?

nebgnahz commented 8 years ago

I went with the background thread option (https://github.com/damellis/ESP/commit/9c522d4ea2bab26e670c3ff0b684949524b683d2).

Quote from my commit:

Not heavily tested yet.

Current implementation is sort of brittle as the mutex protections are spread around. A better idea should be making the class thread-safe; or providing an atomic swap method instead doing copy.

damellis commented 8 years ago

This appears to cause crashes for examples using leave-one-out scoring (e.g. color sensor example). Because scoring happens in a separate thread, training samples are being added and removed from the training data manager while the UI is being rendered. This leads to an assertion failure in the training data manger:

#4  0x00000001000cff8a in TrainingDataManager::getNumSampleForLabel(unsigned int) at /Users/mellis/Source/ESP/Xcode/ESP/src/training-data-manager.cpp:102
#5  0x000000010002bbdf in ofApp::drawTrainingInfo() at /Users/mellis/Source/ESP/Xcode/ESP/src/ofApp.cpp:1508
#6  0x0000000100029e1f in ofApp::draw() at /Users/mellis/Source/ESP/Xcode/ESP/src/ofApp.cpp:1313
#7  0x00000001000398c5 in ofBaseApp::draw(ofEventArgs&) at /Users/mellis/Source/ESP/Xcode/ESP/../../third-party/openFrameworks/libs/openFrameworks/app/ofBaseApp.h:69
#8  0x0000000100559393 in std::__1::shared_ptr<of::priv::Function<ofEventArgs, std::__1::recursive_mutex> > ofEvent<ofEventArgs, std::__1::recursive_mutex>::make_function<ofBaseApp>(ofBaseApp*, void (ofBaseApp::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&)::operator()(void const*, ofEventArgs&) const [inlined] at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworksCompiled/project/osx/../../../../libs/openFrameworks/events/ofEvent.h:237
#9  0x0000000100559371 in decltype(std::__1::forward<ofBaseApp>(fp)(std::__1::forward<void const*, ofEventArgs&>(fp0))) std::__1::__invoke<std::__1::shared_ptr<of::priv::Function<ofEventArgs, std::__1::recursive_mutex> > ofEvent<ofEventArgs, std::__1::recursive_mutex>::make_function<ofBaseApp>(ofBaseApp*, void (ofBaseApp::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&)&, void const*, ofEventArgs&>(ofBaseApp&&, void const*&&, ofEventArgs&&&) [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__functional_base:416
#10 0x0000000100559371 in bool std::__1::__invoke_void_return_wrapper<bool>::__call<std::__1::shared_ptr<of::priv::Function<ofEventArgs, std::__1::recursive_mutex> > ofEvent<ofEventArgs, std::__1::recursive_mutex>::make_function<ofBaseApp>(ofBaseApp*, void (ofBaseApp::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&)&, void const*, ofEventArgs&>(std::__1::shared_ptr<of::priv::Function<ofEventArgs, std::__1::recursive_mutex> > ofEvent<ofEventArgs, std::__1::recursive_mutex>::make_function<ofBaseApp>(ofBaseApp*, void (ofBaseApp::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&)&&&, void const*&&, ofEventArgs&&&) [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__functional_base:437
#11 0x0000000100559371 in std::__1::__function::__func<std::__1::shared_ptr<of::priv::Function<ofEventArgs, std::__1::recursive_mutex> > ofEvent<ofEventArgs, std::__1::recursive_mutex>::make_function<ofBaseApp>(ofBaseApp*, void (ofBaseApp::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&), std::__1::allocator<std::__1::shared_ptr<of::priv::Function<ofEventArgs, std::__1::recursive_mutex> > ofEvent<ofEventArgs, std::__1::recursive_mutex>::make_function<ofBaseApp>(ofBaseApp*, void (ofBaseApp::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&)>, bool (void const*, ofEventArgs&)>::operator()(void const*&&, ofEventArgs&) at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1437
#12 0x0000000100538a85 in std::__1::function<bool (void const*, ofEventArgs&)>::operator()(void const*, ofEventArgs&) const at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1817
#13 0x00000001005388a8 in of::priv::Function<ofEventArgs, std::__1::recursive_mutex>::notify(void const*, ofEventArgs&) [inlined] at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworksCompiled/project/osx/../../../../libs/openFrameworks/events/ofEvent.h:136
#14 0x0000000100538889 in ofEvent<ofEventArgs, std::__1::recursive_mutex>::notify(void const*, ofEventArgs&) at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworksCompiled/project/osx/../../../../libs/openFrameworks/events/ofEvent.h:303
#15 0x0000000100537941 in void ofNotifyEvent<ofEvent<ofEventArgs, std::__1::recursive_mutex>, ofEventArgs>(ofEvent<ofEventArgs, std::__1::recursive_mutex>&, ofEventArgs&) [inlined] at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworksCompiled/project/osx/../../../../libs/openFrameworks/events/ofEventUtils.h:222
#16 0x0000000100537933 in ofCoreEvents::notifyDraw() at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworks/events/ofEvents.cpp:204
#17 0x00000001005a6026 in ofAppGLFWWindow::draw() at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworks/app/ofAppGLFWWindow.cpp:365
#18 0x00000001005589f1 in ofMainLoop::loopOnce() at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworks/app/ofMainLoop.cpp:140
#19 0x0000000100558903 in ofMainLoop::loop() at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworks/app/ofMainLoop.cpp:121
#20 0x00000001005340d5 in ofRunMainLoop() [inlined] at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworks/app/ofAppRunner.cpp:173
#21 0x00000001005340c8 in ofRunApp(std::__1::shared_ptr<ofBaseApp>) at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworks/app/ofAppRunner.cpp:159
#22 0x000000010053404d in ofRunApp(ofBaseApp*) at /Users/mellis/Source/ESP/third-party/openFrameworks/libs/openFrameworks/app/ofAppRunner.cpp:153
#23 0x000000010000221a in main at /Users/mellis/Source/ESP/Xcode/ESP/src/main.cpp:20

@nebgnahz do you have time to take a shot at fixing this?

damellis commented 8 years ago

Let's go back to single-threaded operation for now.

nebgnahz commented 8 years ago

Temporarily reverted with https://github.com/damellis/ESP/commit/7d2a623a078317908bbb10ba92343b13a3df3caa