AlexanderMihail
commented
3 months ago Table::self_test(int test, int loop, string &s) is a long-running task, which, depending on the loop count, may be infinite. The self_test is one-level recursive. It is sure to hog the SHARD that undertook the HTTP request. This was the prime candidate for async-ifying. In the latest version, depending on Table::mode_async toggle, the path of the self_test is going through a wrapper coroutine:
Turned the long-running operation of self_test to coroutine. Another long-runner is the index compactor. It should also be turned to async. Table get/set/list, etc should probably also go the async way, for good measure.
The seastar SIG35 occurs repeatedly generally during sstring operations (probably requiring reallocation). Got to chase those.
It seems to be relating to executor bloked on a long operation (the database self-tests).
Release console message:
Reactor stalled for 69 ms on shard 0. Backtrace: 0x1604ae 0x11eba0 0x11f106 0x11f2df 0x4251f 0x8df74 0x7700b 0x81988 0x60927 0xa17d1 0xa24af 0xa1c5f 0xa41ef 0x1e596e 0x1c18c0 0x1c20a7 0x1c26d6 0x1c4d8d 0x1c5f30 0x11ad57 0x11b082 0x150375 0x150e6c 0xb09d0 0xb13cf 0x7148c 0x29d8f 0x29e3f 0x73944