search

peaclab / MicroFaaS-worker

FaaS on small, embedded-system-like compute nodes
MIT License
0 stars 1 forks source link

Orchestrator frequently tells VMWorkers to reboot due to unhandled request #19

Open abyrne55 opened 2 years ago

abyrne55 commented 2 years ago

(Post-fix bug report; documenting here for posterity)

Running a VM cluster with the latest code on the refactor branch was frequently producing warnings about unhandled requests, e.g.

Sep 03 12:02:53 beaglebone python3[1762]: INFO:root:Processed results of invocation csmZbi from worker 103
Sep 03 12:02:53 beaglebone python3[1762]: INFO:root:Processed results of invocation ixjaZm from worker 104
Sep 03 12:02:53 beaglebone python3[1762]: INFO:root:Sending pkill to VMWorker104
Sep 03 12:02:53 beaglebone python3[1762]: ERROR:root:VMWorker104 made request but no output events set
Sep 03 12:02:53 beaglebone python3[1762]: WARNING:root:Telling VMWorker104 to reboot due to unhandled request
Sep 03 12:02:53 beaglebone python3[1762]: INFO:root:Transmitted work to VMWorker105
Sep 03 12:02:54 beaglebone python3[1762]: INFO:root:Processed results of invocation FlS9lG from worker 105
Sep 03 12:02:54 beaglebone python3[1762]: INFO:root:Transmitted work to VMWorker106
Sep 03 12:02:54 beaglebone python3[1762]: INFO:root:Processed results of invocation b1oC9x from worker 106
Sep 03 12:02:54 beaglebone python3[1762]: INFO:root:Transmitted work to VMWorker103
Sep 03 12:02:55 beaglebone python3[1762]: INFO:root:Transmitted work to VMWorker105
Sep 03 12:02:55 beaglebone python3[1762]: INFO:root:Processed results of invocation 1JIlkt from worker 105
Sep 03 12:02:55 beaglebone python3[1762]: INFO:root:Processed results of invocation uf84SY from worker 103
Sep 03 12:02:55 beaglebone python3[1762]: INFO:root:Transmitted work to VMWorker106
Sep 03 12:02:55 beaglebone python3[1762]: INFO:root:Processed results of invocation Hh1bw7 from worker 106
Sep 03 12:02:55 beaglebone python3[1762]: INFO:root:Sending pkill to VMWorker106
Sep 03 12:02:55 beaglebone python3[1762]: ERROR:root:VMWorker106 made request but no output events set
Sep 03 12:02:55 beaglebone python3[1762]: WARNING:root:Telling VMWorker106 to reboot due to unhandled request
Sep 03 12:02:56 beaglebone python3[1762]: INFO:root:Transmitted work to VMWorker105
Sep 03 12:02:56 beaglebone python3[1762]: INFO:root:Transmitted work to VMWorker103
Sep 03 12:02:56 beaglebone python3[1762]: INFO:root:Processed results of invocation 1S6Zsc from worker 103
Sep 03 12:02:56 beaglebone python3[1762]: INFO:root:Sending pkill to VMWorker103
Sep 03 12:02:56 beaglebone python3[1762]: ERROR:root:VMWorker103 made request but no output events set
Sep 03 12:02:56 beaglebone python3[1762]: WARNING:root:Telling VMWorker103 to reboot due to unhandled request
Sep 03 12:02:56 beaglebone python3[1762]: INFO:root:Attempting to power up VMWorker106
Sep 03 12:02:57 beaglebone python3[1762]: INFO:root:Processed results of invocation iKHAFI from worker 105
Sep 03 12:02:57 beaglebone python3[1762]: INFO:root:Attempting to power up VMWorker104

This was concerning because it seemed that VMWorkers were being told to reboot unnecessarily right after being (appropriately) pkilled. The most likely explanation, however, was that the VMWorker was able to squeeze one more worker request after the call to pkill job was executed but before the QEMU process was actually terminated, so the orchestrator wasn't expecting this. In other words, this is a harmless (though annoying) bug, as the VM is getting killed no matter what it's being told to do by the orchestrator in its final moments.

This bug should be fixed by 4d3b7d3. Please close this after confirmation.

allenz1120 commented 2 years ago

After updating the workers.py on the orchestrator and testing, we are still experiencing the same unhandled request warning on the VMs