search

jdillenkofer / camera_proxy

MIT License
45 stars 17 forks source link

Kernel: Out of memory #8

Open goldenhub77 opened 3 years ago

goldenhub77 commented 3 years ago

Upon running the server and streaming/recording one camera for 10 minutes the server process crashes due to running out of memory. Upon reviewing the logs I am seeing the python3 main.py process incrementally grows by 200-700MB of memory until the process kills due to running out of system memory. Any thoughts?

kernel: [13104.512298] Out of memory: Killed process 52834 (python3) total-vm:14380312kB, anon-rss:13377752kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:27244kB oom_score_adj:0

goldenhub77 commented 3 years ago

I am also noticing many ERROR:camera:Exception occured: timed out Trying to reconnect the camera stream. It always reconnects successfully, but continues to occur once every 10 seconds to sometimes once every 10 minutes. I am also noticing periodic ERROR:libav.h264:error while decoding MB 109 33, bytestream -16 but those are few and far between.

goldenhub77 commented 3 years ago

@jdillenkofer You can replicate this issue when you open two concurrent streams on the same host and cancel loading one of them.

goldenhub77 commented 3 years ago

@jdillenkofer See below for ERROR:Exception details:

Exception in thread CameraThread: Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/usr/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/camera_proxy/src/camera_stream_manager.py", line 30, in start_camera camera.start(lambda data: decoder.handle_packet(data)) File "/camera_proxy/src/camera.py", line 24, in start self._start_stream(handle_video_stream) File "/camera_proxy/src/camera.py", line 66, in _start_stream

File "/camera_proxy/src/baichuan_control_layer.py", line 178, in recv_packet packet = self._recv_next_udp_packet() File "/camera_proxy/src/baichuan_control_layer.py", line 143, in _recv_next_udp_packet packet = self.udp_layer.recv_packet() File "/camera_proxy/src/baichuan_udp_layer.py", line 112, in recv_packet data, sender = self.socket.recvfrom(4096) socket.timeout: timed out

goldenhub77 commented 3 years ago

@jdillenkofer Directly after the Exception the additional following errors are thrown upon attempt to reconnect:

goldenhub77 commented 3 years ago

ERROR:libav.h264:missing picture in access unit with size 40000 ERROR:libav.h264:No start code is found. ERROR:libav.h264:Error splitting the input into NAL units. [Errno 1094995529] Invalid data found when processing input; last error log: [h264] Error splitting the input into NAL units.

jdillenkofer commented 3 years ago

I don't know yet what those libav errors are, i have them too. I think the program sometimes reads invalid data into the h264 stream. Not sure where and why this happens.

But I found and fixed the memory leak. The problem was that i didn't remove the frame_callback from the decoder when a stream is cancelled. It is now fixed in the newest version in the develop branch.