Ram usage is increasing continuously while running ecal python code

prakhar114 commented 6 months ago

Problem Description

When I try to run ecal webcam_grabber.py from ecal-camera-samples/ecal_camera_send_python , the ram keeps on increasing and then if i keep running it for longer time,ram gets filled up completely.

Ram usage before running the webcam_grabber.py

Ram usage after running webcam_grabber.py for 5 min

ecal monitor

Protobuff packet size is ~0.6MB

How to reproduce

clone this repo https://github.com/eclipse-ecal/ecal-camera-samples follow the python setup code in Readme connect a full hd(1920,1080) camera via usb (/dev/video0) run the webcam_grabber.py --compressed

Protobuff packet size is ~0.6MB

How did you get eCAL?

Download from Release Page

Environment

eCAL v5.12.5
OS Ubuntu 22.04.4 LTS
Python 3.10.12
OpenCV version 4.9.0

eCAL System Information

------------------------- SYSTEM ---------------------------------
Version                  : v5.12.5 (2024-04-24 14:43:08 +0200)
Platform                 : linux

------------------------- CONFIGURATION --------------------------
Default INI              : /etc/ecal/ecal.ini

------------------------- NETWORK --------------------------------
Host name                : car
Host group name          : car
Network mode             : local
Network ttl              : 2
Network sndbuf           : 5 MByte
Network rcvbuf           : 5 MByte
Multicast cfg version    : v1
Multicast group          : 239.0.0.1
Multicast mask           : 0.0.0.15
Multicast ports          : 14000 - 14010
Multicast join all IFs   : off
Bandwidth limit (udp)    : not limited

------------------------- TIME -----------------------------------
Synchronization realtime : "ecaltime-localtime"
Synchronization replay   : 
State                    :  synchronized 
Master / Slave           :  Master 
Status (Code)            : "everything is fine." (0)

------------------------- PUBLISHER LAYER DEFAULTS ---------------
Layer Mode INPROC        : off
Layer Mode SHM (ZEROCPY) : auto
Layer Mode TCP           : off
Layer Mode UDP MC        : auto

------------------------- SUBSCRIPTION LAYER DEFAULTS ------------
Layer Mode INPROC        : on
Layer Mode SHM           : on
Layer Mode TCP           : on
Layer Mode UDP MC        : on
Npcap UDP Reciever       : off

------------------------- EXPERIMENTAL ---------------------------
SHM Monitoring           : off
SHM Monitoring (Domain)  : ecal_mon
SHM Monitoring (Queue)   : 1024
Network Monitoring       : on
Drop out-of-order msgs   : off

KerstinKeller commented 6 months ago

Hi, thanks for reporting. First we need to analyse, where this RAM build-up is coming from. It could come from the sample code itself (unlikely, since it's pure Python), from the eCAL binding, from Protobuf, or from opencv which is used to read the webcam images.

We can check pure eCAL samples (e.g. person_snd) to see if that builds up memory. Maybe also the VideoCapture is building up data, and not freeing it.

prakhar114 commented 6 months ago

Hello, I tested the following:-

Ran cpp based person_snd(protobuff), with which there was no ram build up issue.
I closed opencv and tried sending array = np.random.randint(10, size=(1920,1080,3)) with the ecal_snd_python code(protobuff), but the ram buildup issue was still there.

Observation:-

When I am trying to send big proto files over ecal in python, I am facing the ram build up issue, but there is not such issue when i am using cpp to do the same.
Also I did an experiment where I converted the proto message to string before sending and then sent the string using simple eCAL string publisher, In which I noticed there was no ram build up issue.
I also noticed that when the size of protobuf packet is significantly small like np.random.randint(10, size=(10,10,3)), there is no ram build up issue.

prakhar114 commented 5 months ago

@Kerstin-Keller any possible solution/workaround to fix this issue?

Peguen commented 4 months ago

Hi @prakhar114 , sorry for the late answer. I now picked up this issue and wanted to reproduce the problem by

setting up a clean and up-to-date Ubuntu 22.04 LTS virtual machine
installing latest eCAL v5.12.6
cloning the webcam example, creating venv, installing requirements.txt and related eCAL python wheel
building the monitor plugin and installed it

I did start the webcam app in the venv, and let everything run for 15-30 minutes. I could not see any increasing ram usage.

Then I modified the setup, like

matching the protofiles to the plugin to also see the webcam grabbers output
changed eCAL version + wheel to 5.12.5
downgraded to opencv 4.9 (by default requirements.txt installed the 4.10 package) and with that also the numpy version from the latest 2.x to the latest 1.x
ran compressed and non compressed
add in the data field of the proto file a np.random.randint(255, size=(2160,3840,3)) array (started with full hd like in your example)

Everytime with 5-15 minutes of monitoring. (Unfortunately?) There was never an increase in RAM usage visible, monitoring with htop.

So in order to debug your issue, could you please provide

a minimal example of code on how the RAM buildup issue happens for you
the installed packages on your venv provided with "pip list" and "pip list --local" ?

I hope that we can find a solution soon together.

eclipse-ecal / ecal