Open rh-mcermak opened 1 year ago
Few notes - probably belongs to the same area in which tmt
runs a background watchdog for a guest (if enabled) to watch for kernel panics and similar issues. It would need tweakable timeouts, a way to enable/disable, and maybe is not applicable to every provision plugin out there (local
might be moot...)
@psss could we prioritize this, please? Wdyt?
Sorry, this slipped through the cracks. Not sure what could be a realistic timeframe for this as there is already so much stuff on the plate but I've added it on the next hacking session to discuss this.
Prioritizing this would be very much appreciated.
@happz planning to look into this for the next release.
Sounds great! Thank you, guys.
So this will be implemented based on checks, user will be required to ask for this feature.
So, kicking things off in https://github.com/teemtee/tmt/pull/2412. There's still work needed, probably will not be ready for 1.29, but the next release, 1.30 at the beginning of December, sounds doable to me.
For now, a thread is spawned which pings the guest running a test. There's logging and a few controls, but no reboots.
@rh-mcermak would you mind sharing your current or common setup for this feature? I.e. how often do your tools ping the box, how many packets, and how long it takes for the check to decide to reboot the box, in terms of ping attempts and time? Are there any caveats, any exceptions, e.g. situations in which there's no response for allocated time and yet the machine is not expected to be rebooted?
I have very simple setups, suitable for tests, and I'd like to have at least one real-life use case, to set some reasonable defaults, and test the use case and a few corner cases. To learn how you use this type of feature.
Hello sir, thanks a ton for looking into it! I'm giving it a one packet ping attempt every minute, and I keep track of last_successful_ping for every test box. If, at any time, the last_successful_ping is older than an hour, reboot. Something like this :)
Update: proposed #2469. Watchdog runs in its own thread, while another thread runs the test (e.g. SSH which then runs test.sh
wrapper on a guest). Watchdog can detect the guest is not responding to pings and/or SSH connection, and may issue a reboot. This action needs to be communicated to the thread managing the test - either that the reboot has been issued, or that the reboot is needed. And there is no place through which these two threads could exchange information, there is no single data structure shareable by both parties. Code passes (plugin, test, guest) tuple between methods, and it makes no sense to add a fourth item. On the other hand, if we bundle these three into a single structure, we can then dedicate a key in this structure, to hold the "reboot required" flag, for example. The flag would be mostly ignored, and it would remain untouched, but the watchdog check would use it to notify the test thread it's time to act.
I'm going to file a couple of issues to track what's needed to support hard reboot and/or reboot without a working SSH connection. It's the missing piece I have not touched yet, but starting with #2512.
Bumping to 1.31. It's not forgotten, it's not dead, it's still moving but there is too much work.
Bumping to 1.32. Necessary components are being merged, but I'm pretty sure there would be no time left to review the main patch properly before 1.31 deadline :/
I've been testing commit b019d65895c38b51856fae3c291c9a55327e5310 and it seemt to test fine. One important factor affecting how reboot works after a kernel crash is kdump service. It, by default, reboots a system after a kernel panic. To let TMT reboot the machine on its own, kdump service needs to be disabled (systemctl stop kdump). The ppc64le systemts are specific in that they don't use kdump. Instead they use fadump. Fadump is not a service that could be stopped. Instead it runs in the firmware of the test system. To suppress a reboot driven by kdump, --kernel-options='panic=0' --kernel-options-post='panic=0' can be used with beaker client command line. However what happens next is that the system panics, but it still responds to pings (!). To detect such state, ping isn't a sufficient detection method. An attempt to ssh in, for instance, seems to work better.
Another thing coming to my mind is that a system can simply fail to reboot. Typical problem is that after a reboot, a system boots into a differenc OS compared to what it was running before.
Another problam might be that user/owner will change after the reboot.
Not saying that TMT should handle all this, just brainstorming ;)
Thanks a ton for implementing the feature! I'd love to start moving to it :)
For aggressive testing, we need a way to reboot an unresponsive test box, so that the testing can resume after reboot. At the moment this functionality is in beaker-jobwatch (--reboot). I need to replicate this in TMT.
Related issues/PRs
artemis
https://gitlab.com/testing-farm/artemis/-/issues/216connect
https://github.com/teemtee/tmt/pull/2512local
mrack
https://github.com/teemtee/tmt/pull/2591podman
testcloud