Killing salt process can cause Qubes to enter invalid state, leaving zombie VM until manual `virsh undefine`

DemiMarie commented 6 years ago

Qubes OS version:

R4.0

Affected component(s):

qubes-core-admin

Steps to reproduce the behavior:

Kill a saltstack process at just the wrong time. The problem is intermittent.

Expected behavior:

All domains that were started are cleaned up

Actual behavior:

Qubes (rarely) gets into an invalid state, in which qubesd and libvirt disagree as to whether a certain domain exists:

qubesd does not think it exists, so it cannot be removed with Qubes tools
but libvirt thinks it exists, so it refuses qubesd’s attempt to create a new vm with the same name

This forced me to do virsh undefine in dom0, which should never be required.

General notes:

Reproducing this bug requires winning a race condition, which is hard.

Related issues:

andrewdavidwong commented 6 years ago

Possibly related: #2112, #1910

DemiMarie commented 6 years ago

The problem is that both libvirtd and qubesd are stateful, and both believe their state to be authoritative. One of them needs to actually be authoritative, and update the state of the other one when needed.

DemiMarie commented 5 years ago

Proposed fix: never remove a domain from qubes.xml until libvirt confirms its removal.

github-actions[bot] commented 1 year ago

This issue is being closed because:

This issue is on the "Release 4.0 updates" milestone.
Qubes OS 4.0 reached EOL (end-of-life) over one year ago.
There has not been any activity on this issue in over one year.

If anyone believes that this issue should be reopened and reassigned to an active milestone, please leave a brief comment. (For example, if a bug still affects Qubes OS 4.1, then the comment "Affects 4.1" will suffice.)

QubesOS / qubes-issues