search

cneira / firecracker-task-driver

nomad task driver that uses firecracker to start micro-vms
Apache License 2.0
148 stars 18 forks source link

Bug- Veth is not releasing when the MicroVM restart #22

Open hitendra-galaxyweblinks opened 2 years ago

hitendra-galaxyweblinks commented 2 years ago

I am working on a project where I am deploying the micro VM using nomad, the driver is working fine but there is an issue, when the VM is restarting or when we are updating the job with new rootfs, the VM is failing to start. When I dug more, I found the driver is unable to assign IP to the new VM as the IP range is exhausted. When I troubleshoot more I found that there are so many Firecracker VM is created, with each restart it provision more and more VMs uncontrollably and exhaust the whole IP range. Kindly refer to the screenshot to support my case. I am still trying to figure out this behavior of the driver. Technically it should update the rootfs and restart the VM with new rootfs and assign the same IP or new one but why it is creating the VM in the background? I would really appreciate the help here.

image image
hitendra-galaxyweblinks commented 2 years ago 
2022-09-09T03:08:38.283+0530 [DEBUG] client.driver_mgr: plugin process exited: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312468 error="exit status 2"
2022-09-09T03:08:38.283+0530 [WARN]  client.driver_mgr: failed to reattach to plugin, starting new instance: driver=firecracker-task-driver err="singleton plugin exited"
2022-09-09T03:08:38.283+0530 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy backoff=5s error="plugin is shut down"
2022-09-09T03:08:38.283+0530 [DEBUG] client.alloc_runner.task_runner: driver plugin has shutdown; attempting to recover task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:38.283+0530 [DEBUG] client.driver_mgr: starting plugin: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver args=["/root/plugins/firecracker-task-driver"]
2022-09-09T03:08:38.283+0530 [WARN]  client.driver_mgr: received fingerprint error from driver: driver=firecracker-task-driver error="plugin is shut down"
2022-09-09T03:08:38.283+0530 [WARN]  client: dropping node event: node_event="Message: ready, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:36.384981362 +0530 IST m=+231.761923755"
2022-09-09T03:08:38.283+0530 [DEBUG] client.driver_mgr: plugin started: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312569
2022-09-09T03:08:38.283+0530 [DEBUG] client.driver_mgr: waiting for RPC address: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver
2022-09-09T03:08:38.300+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: plugin address: driver=firecracker-task-driver address=/tmp/plugin666233150 network=unix timestamp="2022-09-09T03:08:38.300+0530"
2022-09-09T03:08:38.300+0530 [DEBUG] client.driver_mgr: using plugin: driver=firecracker-task-driver version=2
2022-09-09T03:08:38.301+0530 [DEBUG] client.driver_mgr.stdio: received EOF, stopping recv loop: driver=firecracker-task-driver err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-09-09T03:08:38.302+0530 [INFO]  client.driver_mgr.firecracker-task-driver: buildFingerprint(): driver=firecracker-task-driver @module=firecracker-task-driver driver.FingerPrint=healthy timestamp="2022-09-09T03:08:38.301+0530"
2022-09-09T03:08:38.302+0530 [WARN]  client: dropping node event: node_event="Message: failed to fingerprint driver, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:36.701565273 +0530 IST m=+232.078507666"
2022-09-09T03:08:38.316+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Starting firecracker: driver=firecracker-task-driver @module=firecracker-task-driver driver_initialize_container="&{/usr/local/bin/firecracker /tmp/hello-vmlinux.bin  console=ttyS0 reboot=k panic=1 pci=off nomodules /tmp/nginx_v1.ext4  [] galaxy {   []} []    false 1  300    false false [] <nil> 0xd248c0}+" timestamp="2022-09-09T03:08:38.316+0530"

2022-09-09T03:08:38.650+0530 [DEBUG] http: request complete: method=GET path=/v1/client/allocation/de27a9ad-ea65-9ee5-79f3-dc02f224bab6/stats duration="546.022µs"
2022-09-09T03:08:38.659+0530 [DEBUG] client: state changed, updating node and re-registering
2022-09-09T03:08:38.674+0530 [INFO]  client: node registration complete
2022-09-09T03:08:38.674+0530 [DEBUG] client: evaluations triggered by node registration: num_evals=1
2022-09-09T03:08:38.674+0530 [DEBUG] worker: dequeued evaluation: worker_id=3c7c0d84-8f8d-cdec-9d23-f0b43b1fb604 eval_id=d5dc9451-1843-b63d-20fb-f53e86ddda7d type=service namespace=default job_id=Web node_id=baa9672e-99e1-31ec-02b0-7d44e7c945a5 triggered_by=node-update
2022-09-09T03:08:38.674+0530 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=d5dc9451-1843-b63d-20fb-f53e86ddda7d job_id=Web namespace=default worker_id=3c7c0d84-8f8d-cdec-9d23-f0b43b1fb604

results= | Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) (disconnect 0) (reconnect 0) | Desired Changes for "Nginx": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)

2022-09-09T03:08:38.674+0530 [DEBUG] worker.service_sched: setting eval status: eval_id=d5dc9451-1843-b63d-20fb-f53e86ddda7d job_id=Web namespace=default worker_id=3c7c0d84-8f8d-cdec-9d23-f0b43b1fb604 status=complete
2022-09-09T03:08:38.676+0530 [DEBUG] worker: updated evaluation: worker_id=3c7c0d84-8f8d-cdec-9d23-f0b43b1fb604 eval="<Eval \"d5dc9451-1843-b63d-20fb-f53e86ddda7d\" JobID: \"Web\" Namespace: \"default\">"
2022-09-09T03:08:38.676+0530 [DEBUG] worker: ack evaluation: worker_id=3c7c0d84-8f8d-cdec-9d23-f0b43b1fb604 eval_id=d5dc9451-1843-b63d-20fb-f53e86ddda7d type=service namespace=default job_id=Web node_id=baa9672e-99e1-31ec-02b0-7d44e7c945a5 triggered_by=node-update
2022-09-09T03:08:38.694+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Writing to: driver=firecracker-task-driver @module=firecracker-task-driver driver_initialize_container="/tmp/Galaxy-de27a9ad-ea65-9ee5-79f3-dc02f224bab6+" timestamp="2022-09-09T03:08:38.694+0530"
2022-09-09T03:08:38.695+0530 [DEBUG] client.alloc_runner.task_runner: task successfully recovered on driver: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:38.698+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: panic: runtime error: invalid memory address or nil pointer dereference: driver=firecracker-task-driver
2022-09-09T03:08:38.698+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xca7967]: driver=firecracker-task-driver
2022-09-09T03:08:38.698+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: driver=firecracker-task-driver
2022-09-09T03:08:38.698+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: goroutine 70 [running]:: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink.(*Handle).LinkDel(0x18a3cc0, 0x0, 0x0, 0x0, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink/link_linux.go:1351 +0x37: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink.LinkDel(...): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink/link_linux.go:1344: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/driver.(*taskHandle).shutdown(0xc0003140f0, 0x12a05f200, 0x34, 0xc0003140f0): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/driver/handle.go:224 +0x73: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/driver.(*Driver).StopTask(0xc00053e640, 0xc00025a8c0, 0x34, 0x12a05f200, 0x0, 0x0, 0xc00052d500, 0x1143e60): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/driver/driver.go:370 +0x6a: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StopTask(0xc0005c0d20, 0x1130960, 0xc000306d20, 0xc00053ecd0, 0xc0005c0d20, 0xc000306d20, 0xc000074b30): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/server.go:181 +0x8f: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StopTask_Handler(0xf3c3a0, 0xc0005c0d20, 0x1130960, 0xc000306d20, 0xc00052d5c0, 0x0, 0x1130960, 0xc000306d20, 0xc00025a840, 0x3a): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/proto/driver.pb.go:4375 +0x217: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004fcc00, 0x11401e0, 0xc00050a900, 0xc000556000, 0xc00020c9f0, 0x1894718, 0x0, 0x0, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:1024 +0x4f4: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0004fcc00, 0x11401e0, 0xc00050a900, 0xc000556000, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:1313 +0xd97: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000402060, 0xc0004fcc00, 0x11401e0, 0xc00050a900, 0xc000556000): driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:722 +0xbb: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: created by github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).serveStreams.func1: driver=firecracker-task-driver
2022-09-09T03:08:38.699+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:720 +0xa1: driver=firecracker-task-driver
2022-09-09T03:08:38.703+0530 [DEBUG] client.driver_mgr: plugin process exited: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312569 error="exit status 2"
2022-09-09T03:08:38.703+0530 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy backoff=5s error="plugin is shut down"
2022-09-09T03:08:38.703+0530 [DEBUG] client.alloc_runner.task_runner: driver plugin has shutdown; attempting to recover task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:38.703+0530 [WARN]  client.driver_mgr: received fingerprint error from driver: driver=firecracker-task-driver error="plugin is shut down"
2022-09-09T03:08:38.703+0530 [WARN]  client.driver_mgr: failed to reattach to plugin, starting new instance: driver=firecracker-task-driver err="singleton plugin exited"
2022-09-09T03:08:38.704+0530 [DEBUG] client.driver_mgr: starting plugin: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver args=["/root/plugins/firecracker-task-driver"]
2022-09-09T03:08:38.703+0530 [WARN]  client: dropping node event: node_event="Message: ready, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:36.728176949 +0530 IST m=+232.105119340"
2022-09-09T03:08:38.704+0530 [DEBUG] client.driver_mgr: plugin started: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312666
2022-09-09T03:08:38.704+0530 [DEBUG] client.driver_mgr: waiting for RPC address: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver
2022-09-09T03:08:38.741+0530 [DEBUG] client.driver_mgr: using plugin: driver=firecracker-task-driver version=2
2022-09-09T03:08:38.741+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: plugin address: driver=firecracker-task-driver address=/tmp/plugin339428083 network=unix timestamp="2022-09-09T03:08:38.741+0530"
2022-09-09T03:08:38.742+0530 [DEBUG] client.driver_mgr.stdio: received EOF, stopping recv loop: driver=firecracker-task-driver err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-09-09T03:08:38.742+0530 [INFO]  client.driver_mgr.firecracker-task-driver: buildFingerprint(): driver=firecracker-task-driver @module=firecracker-task-driver driver.FingerPrint=healthy timestamp="2022-09-09T03:08:38.742+0530"
2022-09-09T03:08:38.742+0530 [WARN]  client: dropping node event: node_event="Message: failed to fingerprint driver, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:37.021332688 +0530 IST m=+232.398275099"
2022-09-09T03:08:38.744+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Starting firecracker: driver=firecracker-task-driver @module=firecracker-task-driver driver_initialize_container="&{/usr/local/bin/firecracker /tmp/hello-vmlinux.bin  console=ttyS0 reboot=k panic=1 pci=off nomodules /tmp/nginx_v1.ext4  [] galaxy {   []} []    false 1  300    false false [] <nil> 0xd248c0}+" timestamp="2022-09-09T03:08:38.744+0530"

2022-09-09T03:08:39.062+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Writing to: driver=firecracker-task-driver @module=firecracker-task-driver driver_initialize_container="/tmp/Galaxy-de27a9ad-ea65-9ee5-79f3-dc02f224bab6+" timestamp="2022-09-09T03:08:39.062+0530"
2022-09-09T03:08:39.062+0530 [DEBUG] client.alloc_runner.task_runner: task successfully recovered on driver: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: panic: runtime error: invalid memory address or nil pointer dereference: driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xca7967]: driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: goroutine 144 [running]:: driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink.(*Handle).LinkDel(0x18a3cc0, 0x0, 0x0, 0x0, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink/link_linux.go:1351 +0x37: driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink.LinkDel(...): driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink/link_linux.go:1344: driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/driver.(*taskHandle).shutdown(0xc0003ac0f0, 0x12a05f200, 0x34, 0xc0003ac0f0): driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/driver/handle.go:224 +0x73: driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/driver.(*Driver).StopTask(0xc00052a6e0, 0xc0000e4d00, 0x34, 0x12a05f200, 0x0, 0x0, 0xc00054f200, 0x1143e60): driver=firecracker-task-driver
2022-09-09T03:08:39.064+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/driver/driver.go:370 +0x6a: driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StopTask(0xc0005a6ec0, 0x1130960, 0xc00063a2d0, 0xc00052b540, 0xc0005a6ec0, 0xc00063a2d0, 0xc00007bb30): driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/server.go:181 +0x8f: driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StopTask_Handler(0xf3c3a0, 0xc0005a6ec0, 0x1130960, 0xc00063a2d0, 0xc00054f2c0, 0x0, 0x1130960, 0xc00063a2d0, 0xc0000e4cc0, 0x3a): driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/proto/driver.pb.go:4375 +0x217: driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc00009b200, 0x11401e0, 0xc0004c0a80, 0xc0004ac000, 0xc000525a10, 0x1894718, 0x0, 0x0, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:1024 +0x4f4: driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00009b200, 0x11401e0, 0xc0004c0a80, 0xc0004ac000, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:1313 +0xd97: driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003c0240, 0xc00009b200, 0x11401e0, 0xc0004c0a80, 0xc0004ac000): driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:722 +0xbb: driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: created by github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).serveStreams.func1: driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:720 +0xa1: driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr: plugin process exited: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312666 error="exit status 2"
2022-09-09T03:08:39.065+0530 [WARN]  client.driver_mgr: received fingerprint error from driver: driver=firecracker-task-driver error="plugin is shut down"
2022-09-09T03:08:39.065+0530 [WARN]  client.driver_mgr: failed to reattach to plugin, starting new instance: driver=firecracker-task-driver err="singleton plugin exited"
2022-09-09T03:08:39.065+0530 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy backoff=5s error="plugin is shut down"
2022-09-09T03:08:39.065+0530 [DEBUG] client.alloc_runner.task_runner: driver plugin has shutdown; attempting to recover task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:39.065+0530 [DEBUG] client.driver_mgr: starting plugin: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver args=["/root/plugins/firecracker-task-driver"]
2022-09-09T03:08:39.066+0530 [WARN]  client: dropping node event: node_event="Message: ready, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:37.051011199 +0530 IST m=+232.427953591"
2022-09-09T03:08:39.066+0530 [DEBUG] client.driver_mgr: plugin started: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312767
2022-09-09T03:08:39.066+0530 [DEBUG] client.driver_mgr: waiting for RPC address: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver
2022-09-09T03:08:39.080+0530 [DEBUG] client.driver_mgr: using plugin: driver=firecracker-task-driver version=2
2022-09-09T03:08:39.080+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: plugin address: driver=firecracker-task-driver address=/tmp/plugin327846829 network=unix timestamp="2022-09-09T03:08:39.080+0530"
2022-09-09T03:08:39.081+0530 [DEBUG] client.driver_mgr.stdio: received EOF, stopping recv loop: driver=firecracker-task-driver err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-09-09T03:08:39.081+0530 [INFO]  client.driver_mgr.firecracker-task-driver: buildFingerprint(): driver=firecracker-task-driver @module=firecracker-task-driver driver.FingerPrint=healthy timestamp="2022-09-09T03:08:39.081+0530"
2022-09-09T03:08:39.082+0530 [WARN]  client: dropping node event: node_event="Message: failed to fingerprint driver, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:37.303751974 +0530 IST m=+232.680694369"
2022-09-09T03:08:39.082+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Starting firecracker: driver=firecracker-task-driver driver_initialize_container="&{/usr/local/bin/firecracker /tmp/hello-vmlinux.bin  console=ttyS0 reboot=k panic=1 pci=off nomodules /tmp/nginx_v1.ext4  [] galaxy {   []} []    false 1  300    false false [] <nil> 0xd248c0}+" @module=firecracker-task-driver timestamp="2022-09-09T03:08:39.082+0530"
2022-09-09T03:08:39.441+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Writing to: driver=firecracker-task-driver @module=firecracker-task-driver driver_initialize_container="/tmp/Galaxy-de27a9ad-ea65-9ee5-79f3-dc02f224bab6+" timestamp="2022-09-09T03:08:39.440+0530"
2022-09-09T03:08:39.441+0530 [DEBUG] client.alloc_runner.task_runner: task successfully recovered on driver: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: panic: runtime error: invalid memory address or nil pointer dereference: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xca7967]: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: goroutine 129 [running]:: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink.(*Handle).LinkDel(0x18a3cc0, 0x0, 0x0, 0x0, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink/link_linux.go:1351 +0x37: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink.LinkDel(...): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink/link_linux.go:1344: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/driver.(*taskHandle).shutdown(0xc00064e000, 0x12a05f200, 0x34, 0xc00064e000): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/driver/handle.go:224 +0x73: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/driver.(*Driver).StopTask(0xc0001f0780, 0xc00003ed40, 0x34, 0x12a05f200, 0x0, 0x0, 0xc00038a900, 0x1143e60): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/driver/driver.go:370 +0x6a: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StopTask(0xc0001e0460, 0x1130960, 0xc000367740, 0xc000584190, 0xc0001e0460, 0xc000367740, 0xc000621b30): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/server.go:181 +0x8f: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StopTask_Handler(0xf3c3a0, 0xc0001e0460, 0x1130960, 0xc000367740, 0xc00038a9c0, 0x0, 0x1130960, 0xc000367740, 0xc00003ecc0, 0x3a): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/proto/driver.pb.go:4375 +0x217: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc00009ac00, 0x11401e0, 0xc00009b680, 0xc0001eca00, 0xc00017fe30, 0x1894718, 0x0, 0x0, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:1024 +0x4f4: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00009ac00, 0x11401e0, 0xc00009b680, 0xc0001eca00, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:1313 +0xd97: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003bb030, 0xc00009ac00, 0x11401e0, 0xc00009b680, 0xc0001eca00): driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:722 +0xbb: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: created by github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).serveStreams.func1: driver=firecracker-task-driver
2022-09-09T03:08:39.445+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:720 +0xa1: driver=firecracker-task-driver
2022-09-09T03:08:39.459+0530 [DEBUG] client.driver_mgr: plugin process exited: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312767 error="exit status 2"
2022-09-09T03:08:39.459+0530 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy backoff=5s error="plugin is shut down"
2022-09-09T03:08:39.459+0530 [WARN]  client.driver_mgr: failed to reattach to plugin, starting new instance: driver=firecracker-task-driver err="singleton plugin exited"
2022-09-09T03:08:39.459+0530 [DEBUG] client.driver_mgr: starting plugin: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver args=["/root/plugins/firecracker-task-driver"]
2022-09-09T03:08:39.459+0530 [WARN]  client.driver_mgr: received fingerprint error from driver: driver=firecracker-task-driver error="plugin is shut down"
2022-09-09T03:08:39.459+0530 [DEBUG] client.alloc_runner.task_runner: driver plugin has shutdown; attempting to recover task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:39.459+0530 [WARN]  client: dropping node event: node_event="Message: ready, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:37.31985225 +0530 IST m=+232.696794645"
2022-09-09T03:08:39.460+0530 [DEBUG] client.driver_mgr: plugin started: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312866
2022-09-09T03:08:39.460+0530 [DEBUG] client.driver_mgr: waiting for RPC address: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver
2022-09-09T03:08:39.497+0530 [DEBUG] client.driver_mgr: using plugin: driver=firecracker-task-driver version=2
2022-09-09T03:08:39.497+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: plugin address: driver=firecracker-task-driver address=/tmp/plugin062134114 network=unix timestamp="2022-09-09T03:08:39.496+0530"
2022-09-09T03:08:39.497+0530 [DEBUG] client.driver_mgr.stdio: received EOF, stopping recv loop: driver=firecracker-task-driver err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-09-09T03:08:39.498+0530 [INFO]  client.driver_mgr.firecracker-task-driver: buildFingerprint(): driver=firecracker-task-driver @module=firecracker-task-driver driver.FingerPrint=healthy timestamp="2022-09-09T03:08:39.498+0530"
2022-09-09T03:08:39.498+0530 [WARN]  client: dropping node event: node_event="Message: failed to fingerprint driver, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:37.659460858 +0530 IST m=+233.036403268"
2022-09-09T03:08:39.499+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Starting firecracker: driver=firecracker-task-driver @module=firecracker-task-driver driver_initialize_container="&{/usr/local/bin/firecracker /tmp/hello-vmlinux.bin  console=ttyS0 reboot=k panic=1 pci=off nomodules /tmp/nginx_v1.ext4  [] galaxy {   []} []    false 1  300    false false [] <nil> 0xd248c0}+" timestamp="2022-09-09T03:08:39.499+0530"
2022-09-09T03:08:39.704+0530 [DEBUG] client: state changed, updating node and re-registering
2022-09-09T03:08:39.705+0530 [INFO]  client: node registration complete
2022-09-09T03:08:39.705+0530 [DEBUG] client: evaluations triggered by node registration: num_evals=1
2022-09-09T03:08:39.705+0530 [DEBUG] worker: dequeued evaluation: worker_id=60296042-594e-0269-8f0e-0ae405013d1d eval_id=cdd595a5-fc10-69b8-a0be-51138c23a244 type=service namespace=default job_id=Web node_id=baa9672e-99e1-31ec-02b0-7d44e7c945a5 triggered_by=node-update
2022-09-09T03:08:39.705+0530 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=cdd595a5-fc10-69b8-a0be-51138c23a244 job_id=Web namespace=default worker_id=60296042-594e-0269-8f0e-0ae405013d1d

results= | Total changes: (place 0) (destructive 0) (inplace 0) (stop 0) (disconnect 0) (reconnect 0) | Desired Changes for "Nginx": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)

2022-09-09T03:08:39.706+0530 [DEBUG] worker.service_sched: setting eval status: eval_id=cdd595a5-fc10-69b8-a0be-51138c23a244 job_id=Web namespace=default worker_id=60296042-594e-0269-8f0e-0ae405013d1d status=complete
2022-09-09T03:08:39.706+0530 [DEBUG] worker: updated evaluation: worker_id=60296042-594e-0269-8f0e-0ae405013d1d eval="<Eval \"cdd595a5-fc10-69b8-a0be-51138c23a244\" JobID: \"Web\" Namespace: \"default\">"
2022-09-09T03:08:39.706+0530 [DEBUG] worker: ack evaluation: worker_id=60296042-594e-0269-8f0e-0ae405013d1d eval_id=cdd595a5-fc10-69b8-a0be-51138c23a244 type=service namespace=default job_id=Web node_id=baa9672e-99e1-31ec-02b0-7d44e7c945a5 triggered_by=node-update
2022-09-09T03:08:39.909+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Writing to: driver=firecracker-task-driver driver_initialize_container="/tmp/Galaxy-de27a9ad-ea65-9ee5-79f3-dc02f224bab6+" @module=firecracker-task-driver timestamp="2022-09-09T03:08:39.909+0530"
2022-09-09T03:08:39.909+0530 [DEBUG] client.alloc_runner.task_runner: task successfully recovered on driver: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:39.912+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: panic: runtime error: invalid memory address or nil pointer dereference: driver=firecracker-task-driver
2022-09-09T03:08:39.912+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xca7967]: driver=firecracker-task-driver
2022-09-09T03:08:39.912+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: goroutine 57 [running]:: driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink.(*Handle).LinkDel(0x18a3cc0, 0x0, 0x0, 0x0, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink/link_linux.go:1351 +0x37: driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink.LinkDel(...): driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/vishvananda/netlink/link_linux.go:1344: driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/driver.(*taskHandle).shutdown(0xc0003a40f0, 0x12a05f200, 0x34, 0xc0003a40f0): driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/driver/handle.go:224 +0x73: driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/driver.(*Driver).StopTask(0xc0000d29b0, 0xc0003a28c0, 0x34, 0x12a05f200, 0x0, 0x0, 0xc0005d4000, 0x1143e60): driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/driver/driver.go:370 +0x6a: driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StopTask(0xc00065ba20, 0x1130960, 0xc00050e180, 0xc0002cc140, 0xc00065ba20, 0xc00050e180, 0xc000231b30): driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/server.go:181 +0x8f: driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StopTask_Handler(0xf3c3a0, 0xc00065ba20, 0x1130960, 0xc00050e180, 0xc0005d40c0, 0x0, 0x1130960, 0xc00050e180, 0xc0003a2880, 0x3a): driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/github.com/hashicorp/nomad/plugins/drivers/proto/driver.pb.go:4375 +0x217: driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004e5380, 0x11401e0, 0xc0004e5e00, 0xc0004b4000, 0xc0003d1080, 0x1894718, 0x0, 0x0, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.913+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:1024 +0x4f4: driver=firecracker-task-driver
2022-09-09T03:08:39.914+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0004e5380, 0x11401e0, 0xc0004e5e00, 0xc0004b4000, 0x0): driver=firecracker-task-driver
2022-09-09T03:08:39.914+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:1313 +0xd97: driver=firecracker-task-driver
2022-09-09T03:08:39.914+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004c9450, 0xc0004e5380, 0x11401e0, 0xc0004e5e00, 0xc0004b4000): driver=firecracker-task-driver
2022-09-09T03:08:39.914+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:722 +0xbb: driver=firecracker-task-driver
2022-09-09T03:08:39.914+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: created by github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc.(*Server).serveStreams.func1: driver=firecracker-task-driver
2022-09-09T03:08:39.914+0530 [DEBUG] client.driver_mgr.firecracker-task-driver:     /root/go/src/github.com/cneira/firecracker-task-driver/vendor/google.golang.org/grpc/server.go:720 +0xa1: driver=firecracker-task-driver
2022-09-09T03:08:39.916+0530 [DEBUG] client.driver_mgr: plugin process exited: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312866 error="exit status 2"
2022-09-09T03:08:39.916+0530 [WARN]  client.driver_mgr: received fingerprint error from driver: driver=firecracker-task-driver error="plugin is shut down"
2022-09-09T03:08:39.916+0530 [WARN]  client.driver_mgr: failed to reattach to plugin, starting new instance: driver=firecracker-task-driver err="singleton plugin exited"
2022-09-09T03:08:39.917+0530 [WARN]  client: dropping node event: node_event="Message: ready, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:37.688502153 +0530 IST m=+233.065444546"
2022-09-09T03:08:39.916+0530 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy backoff=5s error="plugin is shut down"
2022-09-09T03:08:39.917+0530 [DEBUG] client.alloc_runner.task_runner: driver plugin has shutdown; attempting to recover task: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:39.917+0530 [DEBUG] client.driver_mgr: starting plugin: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver args=["/root/plugins/firecracker-task-driver"]
2022-09-09T03:08:39.917+0530 [DEBUG] client.driver_mgr: plugin started: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver pid=312965
2022-09-09T03:08:39.917+0530 [DEBUG] client.driver_mgr: waiting for RPC address: driver=firecracker-task-driver path=/root/plugins/firecracker-task-driver
2022-09-09T03:08:39.946+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: plugin address: driver=firecracker-task-driver address=/tmp/plugin998461408 network=unix timestamp="2022-09-09T03:08:39.946+0530"
2022-09-09T03:08:39.946+0530 [DEBUG] client.driver_mgr: using plugin: driver=firecracker-task-driver version=2
2022-09-09T03:08:39.947+0530 [DEBUG] client.driver_mgr.stdio: received EOF, stopping recv loop: driver=firecracker-task-driver err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-09-09T03:08:39.948+0530 [INFO]  client.driver_mgr.firecracker-task-driver: buildFingerprint(): driver=firecracker-task-driver @module=firecracker-task-driver driver.FingerPrint=healthy timestamp="2022-09-09T03:08:39.948+0530"
2022-09-09T03:08:39.948+0530 [WARN]  client: dropping node event: node_event="Message: failed to fingerprint driver, Subsystem: Driver, Details: driver: firecracker-task-driver, Timestamp: 2022-09-09 03:08:37.98126072 +0530 IST m=+233.358203111"
2022-09-09T03:08:39.948+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Starting firecracker: driver=firecracker-task-driver @module=firecracker-task-driver driver_initialize_container="&{/usr/local/bin/firecracker /tmp/hello-vmlinux.bin  console=ttyS0 reboot=k panic=1 pci=off nomodules /tmp/nginx_v1.ext4  [] galaxy {   []} []    false 1  300    false false [] <nil> 0xd248c0}+" timestamp="2022-09-09T03:08:39.948+0530"
2022-09-09T03:08:40.223+0530 [INFO]  client.driver_mgr.firecracker-task-driver: Writing to: driver=firecracker-task-driver @module=firecracker-task-driver driver_initialize_container="/tmp/Galaxy-de27a9ad-ea65-9ee5-79f3-dc02f224bab6+" timestamp="2022-09-09T03:08:40.222+0530"
2022-09-09T03:08:40.223+0530 [DEBUG] client.alloc_runner.task_runner: task successfully recovered on driver: alloc_id=de27a9ad-ea65-9ee5-79f3-dc02f224bab6 task=Galaxy driver=firecracker-task-driver
2022-09-09T03:08:40.225+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: panic: runtime error: invalid memory address or nil pointer dereference: driver=firecracker-task-driver
2022-09-09T03:08:40.225+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xca7967]: driver=firecracker-task-driver
2022-09-09T03:08:40.225+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: driver=firecracker-task-driver
2022-09-09T03:08:40.225+0530 [DEBUG] client.driver_mgr.firecracker-task-driver: goroutine 57 [running]:: driver=firecracker-task-driver
cneira commented 2 years ago

Thanks for the bug report, I'll take a look when I have available time to work on this.

Joffref commented 2 years ago

Bug is due to the following issue that has been solved in firecracker-go-sdk upstream https://github.com/firecracker-microvm/firecracker-go-sdk/issues/428.

Unfortunately, this plugin is not up-to-date with the firecraker-go-sdk upstream.

alexhulbert commented 2 years ago

@Joffref I tried cherry-picking that commit on top of the latest pre-1.0-compatible firecracker-go-sdk and basing the firecracker-task-driver off of it, but I'm still getting the same issue. Is there anything else I need to do to get this integrated with the task driver? My forked firecracker-go-sdk and firecracker-task-driver are available at https://github.com/radixbio/firecracker-go-sdk and https://github.com/radixbio/firecracker-task-driver respectively for context.

Thanks for linking the issue!!

cneira commented 2 years ago

@hitendra-galaxyweblinks were you using the cni-plugins from my fork? if not that the cause of the segfault. Unfortunately by mistake a erased my fork with the changes need a month ago, but luckily seems the current sdk fixed the issue. I'll try to try this during the week should be and easy fix. This is described in https://github.com/cneira/firecracker-task-driver/issues/11 , the thing is I don't have the cni plugins I fixed as I delete them by mistake, but is not so bad is this is fixed on the updated sdk.