Closed ylobankov closed 1 year ago
To reproduce the issue, it's suffice to start tarantool in the interactive mode and then exit. The problem is buried somewhere in the on_shutdown
triggers because disabling them helps.
diff --git a/src/main.cc b/src/main.cc
index 5c7587135915..40b89f6c9368 100644
--- a/src/main.cc
+++ b/src/main.cc
@@ -162,7 +162,7 @@ on_shutdown_f(va_list ap)
while (!is_shutting_down)
fiber_yield();
- if (trigger_fiber_run(&box_on_shutdown_trigger_list, NULL,
+ if (0 && trigger_fiber_run(&box_on_shutdown_trigger_list, NULL,
on_shutdown_trigger_timeout) != 0) {
say_error("on_shutdown triggers failed");
diag_log();
This change also helps...
diff --git a/src/lib/core/fiber.c b/src/lib/core/fiber.c
index 958e4a02f..4ab7bd282 100644
--- a/src/lib/core/fiber.c
+++ b/src/lib/core/fiber.c
@@ -710,6 +710,7 @@ fiber_yield(void)
cord->fiber = callee;
callee->flags = (callee->flags & ~FIBER_IS_READY) | FIBER_IS_RUNNING;
+ __asm__ __volatile__("": : :"memory");
ASAN_START_SWITCH_FIBER(asan_state,
(caller->flags & FIBER_IS_DEAD) == 0,
callee->stack,
This also helps:
diff --git a/src/lib/core/fiber.c b/src/lib/core/fiber.c
index 958e4a02f..3e717d64e 100644
--- a/src/lib/core/fiber.c
+++ b/src/lib/core/fiber.c
@@ -711,7 +711,7 @@ fiber_yield(void)
callee->flags = (callee->flags & ~FIBER_IS_READY) | FIBER_IS_RUNNING;
ASAN_START_SWITCH_FIBER(asan_state,
- (caller->flags & FIBER_IS_DEAD) == 0,
+ true,
callee->stack,
callee->stack_size);
coro_transfer(&caller->ctx, &callee->ctx);
A similar crash: https://github.com/google/sanitizers/issues/189#issuecomment-1007798300
So, what happens when Ctrl+D is pressed:
tarantool> ^D
fiber_recycle(103/interactive)
fiber_yield(0/, FIBER_IS_DEAD) -> 1/sched
==2744649==T0: FakeStack destroyed: 0: 594/8192; 1: 153/4096; 2: 74/2048; 3: 0/1024; 4: 30/512; 5: 0/256; 6: 4/128; 7: 0/64; 8: 87/32; 9: 0/16; 10: 0/8;
fiber_call_impl(1/sched -> 102/on_shutdown)
==2744649==T0: FakeStack created: 0x7f1c5027b000 -- 0x7f1c50800000 stack_size_log: 19; mmapped 5652K, noreserve=0
fiber_call_impl(102/on_shutdown -> 104/trigger_fiber0)
tarantool: src/lib/core/fiber.c:728: void fiber_yield(void): Assertion `asan_state != NULL' failed.
103/interactive
fiber is recycled and fiber_yield()
is called to switch to the scheduler.fiber_yield()
ASAN allocates a fake stack by __asan_stack_malloc_0()
for the stack-use-after-return detection.__sanitizer_start_switch_fiber()
is called with fake_stack_save == NULL
, because the fiber is marked as dead.__sanitizer_start_switch_fiber()
destroys the fake stack.coro_transfer()
switches to the scheduler, which switches to 102/on_shutdown
.trigger_fiber_run()
ASAN allocates a new fake stack at the same addresses as [2].104/trigger_fiber0
is created from the recycled 103/interactive
.104/trigger_fiber0
is started, so now we are just after coro_transfer()
, see [5].fiber_yield()
ASAN tries to access the fake stack and crashes, because it was unmapped at [4] and recreated at [6].With ASAN_START_SWITCH_FIBER(asan_state, true, ...)
it doesn't crash, because the fake stack is not destroyed:
tarantool> ^D
fiber_recycle(103/interactive)
fiber_yield(0/, FIBER_IS_DEAD) -> 1/sched
fiber_call_impl(1/sched -> 102/on_shutdown)
==2750102==T0: FakeStack created: 0x7fb22f771000 -- 0x7fb22fcf6000 stack_size_log: 19; mmapped 5652K, noreserve=0
fiber_call_impl(102/on_shutdown -> 104/trigger_fiber0)
fiber_yield(104/session.shutdown, FIBER_IS_DEAD) -> 102/on_shutdown
fiber_recycle(104/session.shutdown)
fiber_recycle(102/on_shutdown)
fiber_yield(0/, FIBER_IS_DEAD) -> 1/sched
cord_destroy()
TLDR: We said to ASAN that we wouldn't switch back to a fiber, but switched to its recycled incarnation.
Tarantool
Steps to reproduce
Open
.test.mk
file and removerun-luajit-test
target from L112. Save and exit.The command above runs one test only. If you want to get all failures, remove
TEST_RUN_EXTRA_PARAMS=app/fiber.test.lua
from the command.Expected result: Test passed.
Actual result: