grahamwhaley
commented
4 years ago Thanks for the report @Dodan . Would you be able to try a couple of things to see if we can find some more clues...
1) before you quit the container where the failure happened, can you do a dmesg in it to get the kernel logs, to see if there was any OOM for instance (adding an exec -ti dmesg command to your test script may do it.
2) Could you run the failing command under strace, so we can capture a log of the system calls, and then hopefully we can find the failing system call (near the end of the logs), and that might provide clues as to where the error is.
thanks!
devimc
Dodan
Description of problem
Hello! My team and I have been running some benchmarks on some various FaaS technologies using various containerization setups. We have been facing some stability issues when using Kata with Firecracker.
Bellow we have put together a simple scenario where a series of apt update/install commands, inside a kata-fc container running Ubuntu, will lead to the container receiving a sigkill.
We noticed that this behaviour replicates for other long running processes that we use (e.g. python http servers or using the containers shell interactively) but the easiest and most reliable way to replicate this was by running the packet manager.
Can you please have a look?
This is our setup:
This is the script we used to recreate the bug:
Show code:
```bash #!/bin/bash # kill_container.sh POD_ID=`crictl runp -r $1 pod.yaml` READY_STATUS=`crictl pods | grep busybox | grep Ready | wc -l` while [[ $READY_STATUS -ne 1 ]]; do READY_STATUS=`crictl pods | grep busybox | grep Ready | wc -l` done echo $POD_ID echo $READY_STATUS CONT_ID=`crictl create $POD_ID container.yaml pod.yaml` READY_STATUS=`crictl ps -a | grep busybox | grep Created | wc -l` while [[ $READY_STATUS -ne 1 ]]; do READY_STATUS=`crictl ps -a | grep busybox | grep Created | wc -l` done echo $CONT_ID echo $READY_STATUS EXEC_STATUS=`crictl exec -it $CONT_ID apt update` echo $EXEC_STATUS EXEC_STATUS=`crictl exec -it $CONT_ID apt install -y htop` echo $EXEC_STATUS EXEC_STATUS=`crictl exec -it $CONT_ID apt install -y build-essential` echo $EXEC_STATUS RM_STATUS=`crictl stop $CONT_ID && crictl rm $CONT_ID` echo $RM_STATUS RMP_STATUS=`crictl stopp $POD_ID && crictl rmp $POD_ID` echo $RMP_STATUS ```Expected result
Actual result
Environment
These are the pod.yaml and container.yaml files we used:
These are the CRI-O network configuration file and crio.conf file we used:
This is the output of the kata-collect-data.sh script: