search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.48k stars 580 forks source link

Allow seastar task shuffling in release build #13627

Open dotnwat opened 1 year ago

dotnwat commented 1 year ago

In our debug builds Seastar will shuffle the order of task execution via SEASTAR_SHUFFLE_TASK_QUEUE. We can think of this as a form of fuzzing and has in the past revealed some race conditions.

It may be interesting to enable this in our release build because it is generally an order of magnitude (or 2?) faster than the debug build. Being that much faster may already have a large impact on how many inter-leavings are common, and adding shuffling may be useful.

This is a compile-time feature in Seastar. We probably do not want to create an entirely new build type on our side, so it would be nice if this were a runtime flag in Seastar. It may be that it is compile time because Seastar devs did not want extra instructions in the scheduling hot path. As far as upstreaming the changes, we could measure the impact or there are probably some tricks we could do to get the instructions out of the hot path (beyond [[unlikely]]?). In the short term it might just be useful to test it out.

JIRA Link: CORE-1457

dotnwat commented 1 year ago

cc @VladLazar, and I think others have suggested this in the past.