Crash (GCSetHaxeFinalizer) when using sqlite from multiple threads

[jeremyfa]

The crash is not systematic. Could even say pretty rare, but it gets reported from time to time.

I have an app that reads from and writes to a sqlite db. It uses sys.db.Sqlite for that. Most of the time, it works, but it can happen that the app crashes when querying the database. Following is an example of crash log I have. It crashes when trying to write to the database from a background thread.

Some info:

• I am using one SqliteConnection per thread (storing each connection with sys.thread.Tls<Connection>)
• The background thread is a regular haxe thread (not a foreign thread).
• The sqlite database is accessed sometimes from main thread, but most of the time from a background thread
• As far as I know, and from taking a look at HXCPP's sqlite code, sqlite is built with default defines, that means SQLITE_THREADSAFE is 1 and opened connections are expected to be thread safe.
• Every access to the db is protected by a mutex on haxe side, although I don't think that's necessary given that sqlite is expected to be thread safe
• The haxe code interacting with sqlite in my project is there: https://github.com/ceramic-engine/ceramic/blob/master/runtime/src/ceramic/SqliteKeyValue.hx (just added some missing mutex.release() calls in exception catch clauses, but the crashes I'm having don't seem to be related to that anyway).

The crash log on the related (background) thread:

EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x000006520b5cce60
Crashed: Thread
0  **** **** *****                0x29e0a48 std::__1::less<hx::Object*>::operator()(hx::Object* const&, hx::Object* const&) const + 54 (__functional_base:54)
1  **** **** *****                0x29e29f8 std::__1::__map_value_compare<hx::Object*, std::__1::__value_type<hx::Object*, void (*)(Dynamic)>, std::__1::less<hx::Object*>, true>::operator()(hx::Object* const&, std::__1::__value_type<hx::Object*, void (*)(Dynamic)> const&) const + 521 (map:521)
2  **** **** *****                0x29e2b8c std::__1::__tree_node_base<void*>*& std::__1::__tree<std::__1::__value_type<hx::Object*, void (*)(Dynamic)>, std::__1::__map_value_compare<hx::Object*, std::__1::__value_type<hx::Object*, void (*)(Dynamic)>, std::__1::less<hx::Object*>, true>, std::__1::allocator<std::__1::__value_type<hx::Object*, void (*)(Dynamic)> > >::__find_equal<hx::Object*>(std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>*&, hx::Object* const&) + 1975 (__tree:1975)
3  **** **** *****                0x29e2a98 std::__1::pair<std::__1::__tree_iterator<std::__1::__value_type<hx::Object*, void (*)(Dynamic)>, std::__1::__tree_node<std::__1::__value_type<hx::Object*, void (*)(Dynamic)>, void*>*, long>, bool> std::__1::__tree<std::__1::__value_type<hx::Object*, void (*)(Dynamic)>, std::__1::__map_value_compare<hx::Object*, std::__1::__value_type<hx::Object*, void (*)(Dynamic)>, std::__1::less<hx::Object*>, true>, std::__1::allocator<std::__1::__value_type<hx::Object*, void (*)(Dynamic)> > >::__emplace_unique_key_args<hx::Object*, std::__1::piecewise_construct_t const&, std::__1::tuple<hx::Object* const&>, std::__1::tuple<> >(hx::Object* const&, std::__1::piecewise_construct_t const&, std::__1::tuple<hx::Object* const&>&&, std::__1::tuple<>&&) + 2092 (__tree:2092)
4  **** **** *****                0x29d8088 std::__1::map<hx::Object*, void (*)(Dynamic), std::__1::less<hx::Object*>, std::__1::allocator<std::__1::pair<hx::Object* const, void (*)(Dynamic)> > >::operator[](hx::Object* const&) + 1521 (map:1521)
5  **** **** *****                0x29d7eec hx::GCSetHaxeFinalizer(hx::Object*, void (*)(Dynamic)) + 2744 (Immix.cpp:2744)
6  **** **** *****                0x29d97fc _hx_set_finalizer(Dynamic, void (*)(Dynamic)) + 6823 (Immix.cpp:6823)
7  **** **** *****                0x2aadbbc (anonymous namespace)::result::create(sqlite3*, sqlite3_stmt*, String) + 50 (Sqlite.cpp:50)
8  **** **** *****                0x2aadb30 _hx_sqlite_request(Dynamic, String) + 256 (Sqlite.cpp:256)
9  **** **** *****                0x976708 sys::db::_Sqlite::SqliteConnection_obj::request(String) + 77 (SqliteConnection.cpp:77)
10 **** **** *****                0x2126574 sys::db::Connection_obj::request(Dynamic, String) + 22 (Connection.h:22)
11 **** **** *****                0x2125cc0 ceramic::SqliteKeyValue_obj::set(String, String) + 165 (SqliteKeyValue.cpp:165)
12 **** **** *****                0xd3d2e0 backend::IO_obj::saveString(String, String) + 123 (IO.cpp:123)
13 **** **** *****                0x1ec923c tracker::SaveModel_obj::_autoSaveAsKeyAppendInBackground(String, String, String, String, String, String) + 374 (SaveModel.cpp:374)
14 **** **** *****                0x1ecb500 tracker::SaveModel_obj::_autoSaveAsKeyAppend(String, String, String, String, String, String)::_hx_Closure_0::_hx_run() + 339 (SaveModel.cpp:339)
15 **** **** *****                0x1ecb444 tracker::SaveModel_obj::_autoSaveAsKeyAppend(String, String, String, String, String, String)::_hx_Closure_0::__run() + 340 (SaveModel.cpp:340)
16 **** **** *****                0x29bf5c0 Dynamic::operator()() + 304 (Dynamic.h:304)
17 **** **** *****                0x8a6bb0 ceramic::BackgroundQueue_obj::internalRunInBackground() + 83 (BackgroundQueue.cpp:83)
18 **** **** *****                0x8a6c18 ceramic::__BackgroundQueue_objinternalRunInBackground(hx::Object*) + 95 (BackgroundQueue.cpp:95)
19 **** **** *****                0x29c4158 hx::CMemberFunction0::__run() + 48 (DynamicImpl.h:48)
20 **** **** *****                0x29bf5c0 Dynamic::operator()() + 304 (Dynamic.h:304)
21 **** **** *****                0x8bf678 sys::thread::_Thread::HaxeThread_obj::create(Dynamic, bool)::_hx_Closure_0::_hx_run() + 108 (HaxeThread.cpp:108)
22 **** **** *****                0x8bf4a4 sys::thread::_Thread::HaxeThread_obj::create(Dynamic, bool)::_hx_Closure_0::__run() + 126 (HaxeThread.cpp:126)
23 **** **** *****                0x29d5720 hxThreadFunc(void*) + 273 (Thread.cpp:273)
24 libsystem_pthread.dylib        0x2458 _pthread_start + 116
25 libsystem_pthread.dylib        0xab0 thread_start + 8

I'm trying to find out why this happens, but running out of ideas.

Are there things I should check in the project, the code that could make the app crash at this place? The crash being rare, I did not manage to reproduce it locally unfortunately.

Got these crashes on iOS, but it is not specific to any iOS version or device, and can't say for sure if it's specific to iOS or if this could happen on Android as well.

The project was built with HXCPP_CAPTURE_SETJMP, using HXCPP 4.2.1.

Any pointer to try to investigate this would be welcome !

[jeremyfa]

I created a stress-test app that specifically tests this in order to reproduce the issue (performing many read and writes to/from an sqlite database from 2 different thread : the main thread + a background thread).

What I discovered is that the app runs fine if we don't do anything specific. There doesn't seem to be any issue with concurrent reads/writes. Mutexes are working as expected etc...

I did however reproduce the crash above from time to time by simply closing the app (tested with a mac build). It seems to happen when the background thread is doing some work and at the same time the app is closed, and, maybe, some hxcpp stuff is getting cleaned-up?

I didn't check in details what happens exactly when the app gets closed, but I guess I should find a way to ensure that when the app is about to close, background thread work should be stopped as well, before HXCPP runtime is shut down. Not sure how to achieve that though.

[hughsando]

It would be helpful to see the stack traces of all the threads in the case of a crash - especially if there is just 1 other thread. However, it looks like it it crashing the c++ code, which generally means a memory overwrite somewhere. One way to better stress test is to add explicit gc_collect calls in different places - for example randomly in the sql results construction. This can help uncover issues.