search

peaclab / MicroFaaS-worker

FaaS on small, embedded-system-like compute nodes
MIT License
0 stars 1 forks source link

Runaway starting of VMs near end of experiment #17

Open abyrne55 opened 2 years ago

abyrne55 commented 2 years ago

Observing behavior where VM server starts OOM-killing near the end of our experiments due to apparently infinite creation of VMs. Log tail looks like:

Aug 20 11:08:22 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker103
Aug 20 11:08:26 beaglebone python3[490]: INFO:root:Transmitted work to VMWorker105
Aug 20 11:08:26 beaglebone python3[490]: INFO:root:Processed results of invocation drZG71 from worker 105
Aug 20 11:08:28 beaglebone python3[490]: INFO:root:Transmitted work to VMWorker105
Aug 20 11:08:28 beaglebone python3[490]: INFO:root:Processed results of invocation pNUgnz from worker 105
Aug 20 11:08:29 beaglebone python3[490]: INFO:root:Transmitted work to VMWorker105
Aug 20 11:08:29 beaglebone python3[490]: INFO:root:Processed results of invocation LXgBkQ from worker 105
Aug 20 11:08:29 beaglebone python3[490]: INFO:root:Transmitted work to VMWorker108
Aug 20 11:08:29 beaglebone python3[490]: INFO:root:Processed results of invocation 6yEaaU from worker 108
Aug 20 11:08:34 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker112
Aug 20 11:08:40 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker109
Aug 20 11:08:41 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker106
Aug 20 11:08:44 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker115
Aug 20 11:08:45 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker114
Aug 20 11:08:47 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker104
Aug 20 11:08:49 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker116
Aug 20 11:08:50 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker110
Aug 20 11:08:51 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker111
Aug 20 11:08:53 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker113
Aug 20 11:09:09 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker107
Aug 20 11:09:15 beaglebone python3[490]: INFO:root:Transmitted work to VMWorker106
Aug 20 11:09:22 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker103
Aug 20 11:09:34 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker112
Aug 20 11:09:40 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker109
Aug 20 11:09:44 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker115
Aug 20 11:09:45 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker114
Aug 20 11:09:47 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker104
Aug 20 11:09:49 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker116
Aug 20 11:09:50 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker110
Aug 20 11:09:51 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker111
Aug 20 11:09:53 beaglebone python3[490]: INFO:root:Attempting to power up VMWorker113
abyrne55 commented 2 years ago

If c62f62c doesn't fix this bug, consider making VMWorker send a pkill just before every qemu command