Open dvyukov opened 5 years ago
It helped to kill -9 these vmd processes. We probably should automate this killing somehow, or of course fix vmd.
It again stuck in vmctl: start vm command failed: Operation already in progress
dead loop.
@pdvyas, is this commit likely to fix the issue here? https://github.com/openbsd/src/commit/b9c4a1f5649a1ac6b055dfda173622d298750a92
That fix should be unrelated. That commit addresses losing access to a vm when sending (vm migration) fails.
There is a thread on openbsd-misc about this: https://marc.info/?l=openbsd-tech&m=153817959129638&w=2
This appears to be the same issue that I documented fairly extensively in "vmd losing VMs" thread on openbsd-tech. If I had to guess, some IPC between the different vmd processes goes awry. Debugging is fairly frustrating due to my being unable to find any IPC tracing mechanisms.
I think an automated daily reboot is a fine idea due to syz-bot recovering state from disk in short order.
For the 2 cases that I observed, killing a vmd process helped. So if we could map vmctl instance to vmd process pid, then we could kill vmd when closing a VM. A simpler version of this is to run "killall -9 vmd" every n hours.
OpenBSD instance stopped working after syz-ci restart because all vmd VMs are in some dead unrecoverable state. Creation of new VMs always fails:
stop -f
does not kill them:@mptre @blackgnezdo