search

madsim-rs / madsim

Magical Deterministic Simulator for distributed systems in Rust.
Apache License 2.0
622 stars 41 forks source link

How to deal with thread spawned outside of madsim simulation #144

Closed bsbds closed 1 year ago

bsbds commented 1 year ago

Hi, thanks for maintaining this fantastic project! I have a question regarding the scenario of invoking external C libraries that may create system threads.

For instance, I've been using rocksdb on madsim runtime, the main thread of rocksdb operates inside the madsim context while the threads it spawned do not, thereby leading to an inconsistency.

Any suggestions on this? Currently I have to allow spawning a system thread and call rocksdb functions inside it.

wangrunji0408 commented 1 year ago

Hi! Glad to see madsim is being used in Xline.

It is intended to forbid creating system thread in the simulation in order to prevent non-determinism.

We have encountered a similar issue, where a third-party library, moka, created a system thread in the background for cache invalidation. Our solution was to replace it by a BTreeMap in the simulation.

Generally, I would suggest replace third-party libraries by mocks in the simulation, so that we could avoid any potential uncertainty. However, if this library is critical to functionality and you believe the system thread it creates won't break the determinism, I can introduce a new API to temporarily disable this check.

What do you think?

bsbds commented 1 year ago

Thanks for the suggestions! That makes sense. I'll try replacing the rocksdb with a mock one in the simulation.