Closed Dzejrou closed 1 year ago
Thank you for your excellent and detailed analysis. I was able to easily reproduce the problem using putty from a Windows laptop to a Linux system. After spending some time in the debugger I found that the root cause of the problem is that the write callback is re-enabling the read event for /dev/tty. Even though del_io_events() avoids enabling read events itself, if there is buffered data destined for /dev/tty, once that data is written the read event would be enabled and the event loop never completed.
I believe the best fix is to just avoid enabling the read events in the write callback for /dev/tty if the command is no longer running. That will solve the problem at hand while and still completely flush the buffers, even if that takes more than one write. It may also prevent sudo from reading tty input when there is no reason for it to do so. I am currently testing a diff to do this and so far I am unable to reproduce the hang, which previously I could trigger after less than a minute.
In 994f86b40 the flag SUDO_EVLOOP_ONCE was passed to del_io_events when called from from pty_close to avoid potential hangs on non-BSD systems. Later, in 8f694cc22, the flag argument was removed from del_io_events in favor of a boolean argument determining whether the function should block (which was the other flag being passed to it besides SUDO_EVLOOP_ONCE). This change lead to the original hang fix to be lost and the same - or similar - hang is now reproducible in sudo.
This commit reintroduces the flag argument to del_io_events with both the SUDO_EVLOOP_ONCE and SUDO_EVLOOP_NONBLOCK values.
This bug was reported to me as occurring when sudo was (repeatedly) invoked from PuTTY terminal connected to a linux-based system (SLES with sudo-1.9.5p2, but I also reproduced it on openSUSE Tumbleweed with sudo-1.9.12p2) when it receives input (letters + return). For the reporter (windows) it took 2-3 minutes to reproduce the issue, but I managed to do that on a linux-based system under wine after 20+ minutes of non-stop typing, so I assume this is some kind of a race related to buffer flushing.
After debugging this issue I found out that restoring the logic used in 994f86b40 made the issue go away both for me (longer reproduction time, so less reliable as I've seen reproduction times over 1 hour [*]) and the reporter (reliably reproduced in 2-3 minutes, so their feedback made me think this was the same issue as the one fixed by 994f86b40).
Some extra data in case you'd want to choose a different approach and these might be of use to you:
[*] Report from when I finally managed to reproduce on Tumbleweed under wine (just to demonstrate the timings):
The stacktrace of a hanged sudo process:
My reproducer (requires X window system, wine, putty.exe and xdotool):
What it does is that it repeatedly opens PuTTY terminals (the terminal would freeze after several minutes of keyboard input due to how fast it was sent to it, depending on the delay between simulated key presses - ranging from 2 minutes to 20 minutes approximately with different values I tried -- I assume this is ssh/X related) and executes sudo in a loop while sending simulated key presses of a character key and return. This was then watched on the target guest for defunct sudo instances.