udev-video-broker gets OOM error in microk8s

[jbuchacher]

Describe the bug The udev-video-broker pod crashes with an OOMError, even after setting udev.configuration.brokerPod.resources.memoryLimit=400Mi.

Output of kubectl get pods,akrii,akric -o wide

NAME                                             READY   STATUS             RESTARTS        AGE    IP             NODE     NOMINATED NODE   READINESS GATES
pod/akri-video-streaming-app-7f5c4d95cc-zqxwn    1/1     Running            5 (68m ago)     17h    10.1.243.225   ubuntu   <none>           <none>
pod/akri-udev-discovery-daemonset-nrhf9          1/1     Running            3 (68m ago)     167m   10.1.243.248   ubuntu   <none>           <none>
pod/akri-udev-video-19090a-pod                   0/1     Pending            0               65m    <none>         <none>   <none>           <none>
pod/akri-controller-deployment-59564bbd4-cj4l2   1/1     Running            0               64m    10.1.243.237   ubuntu   <none>           <none>
pod/akri-agent-daemonset-n7m5r                   1/1     Running            0               50m    10.1.243.253   ubuntu   <none>           <none>
pod/akri-udev-video-3745ea-pod                   0/1     CrashLoopBackOff   6 (2m14s ago)   17m    10.1.243.244   ubuntu   <none>           <none>

NAME                                      CONFIG            SHARED   NODES        AGE
instance.akri.sh/akri-udev-video-19090a   akri-udev-video   false    ["ubuntu"]   150m
instance.akri.sh/akri-udev-video-3745ea   akri-udev-video   false    ["ubuntu"]   17m

NAME                                    CAPACITY   AGE
configuration.akri.sh/akri-udev-video   1          167m

Kubernetes Version: [e.g. Native Kubernetes 1.19, MicroK8s 1.19, Minikube 1.19, K3s] MicroK8s v1.27.4 revision 5650

To Reproduce Steps to reproduce the behavior:

1. Install microk8s following: https://microk8s.io/docs/install-raspberry-pi
2. Install Akri per instructions in https://docs.akri.sh/demos/usb-camera-demo-rpi4
3. Install command for Akri that was used is:

   microk8s.helm upgrade --install akri akri-helm-charts/akri \
   $AKRI_HELM_CRICTL_CONFIGURATION \
   --set udev.discovery.enabled=true \
   --set udev.configuration.enabled=true \
   --set udev.configuration.name=akri-udev-video \
   --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video0"' \
   --set udev.configuration.brokerPod.image.repository="ghcr.io/project-akri/akri/udev-video-broker" \
   --set udev.configuration.brokerPod.resources.memoryLimit=400Mi

Expected behavior Video broker pods will come up healthy and not be killed with an OOM error.

Logs (please share snips of applicable logs)

• To get the logs of any pod, run kubectl logs <pod name>
• To get the logs of a pod that has already terminated, kubectl get logs <pod name> --previous
• If you believe that the problem is with the Kubelet, run journalctl -u kubelet or journalctl -u snap.microk8s.daemon-kubelet if you are using a MicroK8s cluster.

No entries in journalctl: `-- No entries --

udev-video-broker pod logs:

akri.sh udev_broker ... env_logger::init
akri.sh udev_broker ... env_logger::init finished
[2023-09-01T19:36:56Z INFO  udev_video_broker] akri.sh Udev Broker logging started
[2023-09-01T19:36:56Z TRACE udev_video_broker] get_video_devnode - getting devnode
[2023-09-01T19:36:56Z TRACE udev_video_broker] get_video_devnode - found devnode /dev/video0
[2023-09-01T19:36:56Z TRACE udev_video_broker::util::camera_capturer] build_and_start_camera_capturer - entered
[2023-09-01T19:36:56Z TRACE udev_video_broker::util::camera_capturer] get_format - format not set ... trying to use "MJPG"
[2023-09-01T19:36:56Z TRACE udev_video_broker::util::camera_capturer] get_format - using "MJPG" format
[2023-09-01T19:36:56Z TRACE udev_video_broker::util::camera_capturer] get_env_var_resolution - resolution width not set
[2023-09-01T19:36:56Z TRACE udev_video_broker::util::camera_capturer] get_resolution - camera does not support (640, 480) resolution, using (32, 32) resolution
[2023-09-01T19:36:56Z TRACE udev_video_broker::util::camera_capturer] main - frames per second not set ... trying to use 10

Additional context

• Increasing udev.configuration.brokerPod.resources.memoryLimit to 400Mi did not fix the issue, but it increased the amount of time that the pod is alive before crashing/restarting.
• I can successfully stream video from this /dev/video0 device using GStreamer over UDP outside of Kubernetes/Akri
• I can capture JPEGs with v4l2-ctl --device /dev/video0 --set-fmt-video=width=3280,height=2464,pixelformat=JPEG --stream-mmap --stream-to=frame.jpg --stream-count=1
• When I change that command to use pixelFormat=MJPG it hangs, but I'm not sure if that's expected or not
• Raspberry Pi version is: Raspberry Pi 4 Model B Rev 1.2 with 4GB of RAM. Also tried with 8GB of RAM
• I can't get the avenc_mjpeg plugin to work properly in the test gstreamer pipelines. I get the following error:

  sudo gst-launch-1.0 -v videotestsrc pattern=ball ! "video/x-raw,width=640,height=480,framerate=10/1" ! avenc_mjpeg ! v4l2sink device=/dev/video1
  Setting pipeline to PAUSED ...
  Pipeline is PREROLLING ...
  /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)Y444, width=(int)640, height=(int)480, framerate=(fraction)10/1, multiview-mode=(string)mono, colorimetry=(string)2:4:7:1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
  /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)Y444, width=(int)640, height=(int)480, framerate=(fraction)10/1, multiview-mode=(string)mono, colorimetry=(string)2:4:7:1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
  /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)Y444, width=(int)640, height=(int)480, framerate=(fraction)10/1, multiview-mode=(string)mono, colorimetry=(string)2:4:7:1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
  ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
  Additional debug info:
  ../libs/gst/base/gstbasesrc.c(3132): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
  streaming stopped, reason not-negotiated (-4)
  ERROR: pipeline doesn't want to preroll.
  Setting pipeline to NULL ...
  Freeing pipeline ...

• I'm using Ubuntu Server 64 bit:

  Distributor ID: Ubuntu
  Description:    Ubuntu 23.04
  Release:    23.04
  Codename:   lunar

[github-actions[bot]]

Issue has been automatically marked as stale due to inactivity for 90 days. Update the issue to remove label, otherwise it will be automatically closed.

[github-actions[bot]]

Issue has been automatically marked as stale due to inactivity for 90 days. Update the issue to remove label, otherwise it will be automatically closed.