Historical issues

[woodliu]

https://github.com/weaveworks/ignite/issues/979

[woodliu]

This issue is happend when use ignited to start vm. Can hannped when:

1. Use ignited start vm, and then delete the vm manifest in /etc/firecracker/manifests/, ignited will be crashed
2. Use ignited start vm, and then change the vm manifest spec.status.running to false, ignited will be crashed

About the first case, when you delete the vm manifest file, ignited daemon will clear the vm status.runtime, but it will use it to remove containerd in StopVM, this cause issue

        if upd.Event == update.ObjectEventDelete {
            // As we know this VM was deleted, it wouldn't show up in a Get() call
            // Construct a temporary VM object for passing to the delete function
            vm = &api.VM{
                TypeMeta:   *upd.APIType.GetTypeMeta(),
                ObjectMeta: *upd.APIType.GetObjectMeta(),
                Status: api.VMStatus{
                    Running: true, // TODO: Fix this in StopVM
                },
            }
        } 

And about the second cause, when changing the vm manifest file, there will be an update event, but after that, the file will be deleted automatically because the storage received a delete event. I think this is a bug of github.com/weaveworks/libgitops when it handle the notify.InMovedFrom event in function:

// If the moveCache isn't cancelled, the move is considered incomplete and this
// method is fired. A complete move consists out of a "from" event and a "to" event,
// if only one is received, the file is moved in/out of a watched directory, which
// is treated as a normal creation/deletion by this method.
func (m *moveCache) incomplete() {
    var event FileEvent

    switch m.event.Event() {
    case notify.InMovedFrom:
        event = FileEventDelete
    case notify.InMovedTo:
        event = FileEventModify
    default:
        // This should never happen
        panic(fmt.Sprintf("moveCache: unrecognized event: %v", m.event.Event()))
    }

    log.Tracef("moveCache: Timer expired for %d, dispatching...", m.cookie())
    m.watcher.sendUpdate(&FileUpdate{event, m.event.Path()})

    // Delete the cache after the timer has fired
    delete(moveCaches, m.cookie())
}

Here is a modify file test using github.com/rjeczalik/notify, you can see there is a InMovedFrom for test.yaml, but no InMovedTo in this case, it will consider it is a incomplete move, and delete the file.

event is : notify.InDelete /home/charlie.liu/test1/.test.yaml.swp
event is : notify.InCloseWrite /home/charlie.liu/test1/.test.yaml.swx
event is : notify.InDelete /home/charlie.liu/test1/.test.yaml.swx
event is : notify.InCloseWrite /home/charlie.liu/test1/.test.yaml.swp
event is : notify.InMovedTo /home/charlie.liu/test1/test.yaml~
event is : notify.InCloseWrite /home/charlie.liu/test1/4913
event is : notify.InDelete /home/charlie.liu/test1/4913
event is : notify.InMovedFrom /home/charlie.liu/test1/test.yaml
event is : notify.InCloseWrite /home/charlie.liu/test1/test.yaml
event is : notify.InDelete /home/charlie.liu/test1/test.yaml~
event is : notify.InCloseWrite /home/charlie.liu/test1/.test.yaml.swp
event is : notify.InDelete /home/charlie.liu/test1/.test.yaml.swp

It uses incomplete to decide trigger delete event when move vm manifest file out of /etc/firecracker

Test env： System：AlmaLinux release 9.2 (Turquoise Kodkod) Kernel：5.14.0-284.11.1.el9_2.x86_64