search

subconsciousnetwork / noosphere

Noosphere is a protocol for thought; let's discover it together!
Apache License 2.0
667 stars 40 forks source link

sled::Tree::flush_async() deadlocks: Intermittent test failures #403

Closed jsantell closed 1 year ago

jsantell commented 1 year ago

I've seen this happen occasionally when running tests locally.

test single_player_single_device_end_to_end_workflow has been running for over 60 seconds
test writes_do_not_block_reads has been running for over 60 seconds
jsantell commented 1 year ago

Caused when running multiple tests in rust/noosphere/tests/integration.rs from the failing cases each first call to sphere_channel().mutable().save(None), ultimately calling sphere_context.db_mut().flush(). The initial tokio::join in SphereDb<S>::flush never completes in either test (intermittently).

jsantell commented 1 year ago

Due to sled's flush_async() causing deadlocks. This was fixed in January 2021, but there hasn't been a release since. Using the main branch of sled fixes the intermittents, as I'm unable to repro the failures over a reasonable amount of iterations.