search

weaveworks / ignite

Ignite a Firecracker microVM
https://ignite.readthedocs.org
Apache License 2.0
3.49k stars 228 forks source link

ignited in daemon mode deletes the VM config #668

Open darkowlzz opened 4 years ago

darkowlzz commented 4 years ago

As per https://ignite.readthedocs.io/en/stable/installation/#ignited-daemon, ignited creates VMs declaratively. But ignited seems to delete the VM config from /etc/firecracker/manifests/ after creating a VM:

DEBU[0119] FileWatcher: Sending update: MODIFY -> "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0119] FileWatcher: Dispatched events batch and reset the events cache
DEBU[0119] GenericMappedRawStorage: AddMapping: "vm/7F4A12BE-9ADF-4203-B6F6-5F0F72C8A9AE" -> "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0119] SyncStorage: Received update {{CREATE &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/v1alpha3,}} 0xc00010cf00} true
DEBU[0119] SyncStorage: Sent update: {CREATE &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/v1alpha3,}}
DEBU[0119] FileWatcher: Skipping suspended event MODIFY for path: "/etc/firecracker/manifests/smoke-test.yml"
INFO[0119] Creating VM "7F4A12BE-9ADF-4203-B6F6-5F0F72C8A9AE" with name "smoke-test"...
DEBU[0119] Ensuring image weaveworks/ignite-ubuntu:latest exists, or importing it...
DEBU[0119] Importing image with ociRef "weaveworks/ignite-ubuntu:latest"
DEBU[0119] containerd: Inspecting image "weaveworks/ignite-ubuntu:latest"
INFO[0119] containerd image "weaveworks/ignite-ubuntu:latest" not found locally, pulling...
DEBU[0119] containerd: Pulling image "weaveworks/ignite-ubuntu:latest"
DEBU[0119] resolving                                     host=registry-1.docker.io
DEBU[0119] do request                                    host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.3.3+unknown request.method=HEAD url="https://registry-1.docker.io/v2/weaveworks/ignite-ubuntu/manifests/latest"
DEBU[0120] FileWatcher: Sending update: DELETE -> "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0120] SyncStorage: Received update {{DELETE &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/__internal,}} 0xc00010cf00} true
DEBU[0120] GenericMappedRawStorage: RemoveMapping: "vm/7F4A12BE-9ADF-4203-B6F6-5F0F72C8A9AE"
DEBU[0120] SyncStorage: Sent update: {DELETE &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/__internal,}}
DEBU[0120] FileWatcher: Skipping suspended event DELETE for path: "/etc/firecracker/manifests/smoke-test.yml"
...

The VM is left behind in stopped state:

$ sudo ignite ps -a
VM ID                   IMAGE               KERNEL                  SIZE    CPUS    MEMORY      CREATED     STATUS  IPS     PORTS   NAME
7F4A12BE-9ADF-4203-B6F6-5F0F72C8A9AE    weaveworks/ignite-ubuntu:latest weaveworks/ignite-kernel:4.19.125   3.0 GB  1   800.0 MB    5m49s ago   Stopped 10.61.0.2       smoke-test

Refer https://github.com/weaveworks/ignite/issues/659#issuecomment-674929825 for more logs.

mvgijssel commented 2 years ago

I'm running into a similar problem where the ignited daemon removes the config whenever I make a change to a vm template within the /etc/firecracker/manifests/ directory.

These are the logs of ignited when I try to update the ip address of the vm

Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: time="2022-04-21T07:43:08Z" level=info msg="Removing VM \"599615df99804ae9\" with name \"<deleted>\"..."
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc649f8]
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: goroutine 35 [running]:
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: github.com/weaveworks/ignite/pkg/operations.StopVM(0xc00065f500, 0xc000200101, 0x17, 0xc000671b80)
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/pkg/operations/remove.go:89 +0x78
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: github.com/weaveworks/ignite/pkg/operations.CleanupVM(0xc00065f500, 0x4, 0x155f599)
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/pkg/operations/remove.go:38 +0x2e9
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: github.com/weaveworks/ignite/pkg/operations/reconcile.remove(0xc00065f500, 0x1a8, 0x7f916eca4108)
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:166 +0x11e
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: github.com/weaveworks/ignite/pkg/operations/reconcile.handleDelete(...)
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:104
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: github.com/weaveworks/ignite/pkg/operations/reconcile.ReconcileManifests.func2(0xc000010eb8, 0xc00005a800)
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:74 +0x2a
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: github.com/weaveworks/ignite/pkg/operations/reconcile.runHandle(0xc00017be78)
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:85 +0x2b
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: github.com/weaveworks/ignite/pkg/operations/reconcile.ReconcileManifests(0xc0004784a0)
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:72 +0x606
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: github.com/weaveworks/ignite/cmd/ignited/cmd.NewCmdDaemon.func1.1(0xc0004784a0)
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/cmd/ignited/cmd/daemon.go:32 +0x6a
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]: created by github.com/weaveworks/ignite/cmd/ignited/cmd.NewCmdDaemon.func1
Apr 21 07:43:08 default-ubuntu-focal ignited[3568]:         /go/src/github.com/weaveworks/ignite/cmd/ignited/cmd/daemon.go:30 +0xde
Apr 21 07:43:08 default-ubuntu-focal systemd[1]: ignited.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 21 07:43:08 default-ubuntu-focal systemd[1]: ignited.service: Failed with result 'exit-code'.
Apr 21 07:43:08 default-ubuntu-focal systemd[1]: ignited.service: Scheduled restart job, restart counter is at 6.
Apr 21 07:43:08 default-ubuntu-focal systemd[1]: Stopped Run the ignited daemon.
Apr 21 07:43:08 default-ubuntu-focal systemd[1]: Started Run the ignited daemon.
Apr 21 07:43:08 default-ubuntu-focal ignited[3919]: time="2022-04-21T07:43:08Z" level=info msg="Starting reconciliation loop..."
Apr 21 07:43:08 default-ubuntu-focal ignited[3919]: time="2022-04-21T07:43:08Z" level=error msg="prometheus server was stopped with error: listen unix /var/lib/firecracker/daemon.sock: bind: address already