Memory usage issue

[thenexxuz-zz]

I am trying to use bmdcapture to pipe to an instance of ffmpeg segmented streaming for 24/7 recording but I keep running into the same issue, running out of memory. The server it is running on has 6GB of ram but after about 2 and a half days the ram fills up completely and bmdcapture dies then the ram is cleared. I have confirmed that piping bmdcapture to /dev/null still fills up the ram.

Are there any suggestions?

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/942865-memory-usage-issue?utm_campaign=plugin&utm_content=tracker%2F256764&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F256764&utm_medium=issues&utm_source=github).

[lu-zero]

ffmpeg isn't supported anymore, please use avconv.

make sure you are processing frames in real-time, the capture just piles up frames if they aren't processed fast enough.

[thenexxuz-zz]

I must still have something incorrect... its memory usage still vastly surpassed the 1GB mark.

This is the command that I am using to capture my live stream: /usr/bin/bmdcapture -C 0 -m 12 -A 2 -V 4 -F nut -f pipe:1 2> /data/bmdCard0.txt | avconv -y -i - -vf "yadif" -c:v "mpeg2video" -q:v 2 -s 640x360 -c:a "mp2" -pix_fmt "yuv420p" -map 0 -loglevel fatal -f segment -vol 24576 -segment_time 60 /dev/shm/card0/out%03d.mpg &> /data/captureCard0.txt

Right now the recordings are being saved as one minute clips on ram disk for later processing. For testing I have them immediately deleted with a cron every minute until I can get this memory issue under control.

Am I leaving something out?

[thenexxuz-zz]

I have tried many different iterations of the command that I posted the other day but I still cannot get this to stop the continuous growth.

What do you mean "processing frames in real-time"?

[hoffie]

Unfortunately I can't provide a solution either -- I can just leave a "mee too" here and share some experience. I had the same problem: memory usage just grew over time. During testing, I also tried sending the data to /dev/null which did work for me. So my conclusion was that the encoder was too slow (for whatever reason -- it wasn't maxing out the CPUs). If sending to /dev/null exhibits the same problem for you, it might either be a different issue or a performance problem as well (but even without an encoder).

If the problem itself (encoding is delayed somewhere) cannot be solved, maybe it would be useful to drop frames in bmdcapture. I made some attempts to do so, but failed to drop video and audio frames properly, which led to problems in the encoder (waiting for the related audio frame which would never be sent because it was dropped...).

I have no idea how easy or hard it would be to implement that.

I don't know which libav version you are using, but it might be possible, that some dev version leaked memory (didn't check), so using a release version might be a good idea.

(In my setup I just had to do live streaming for a specific time frame; just upping the bmdcapture memory limit worked for me because of that)

[lu-zero]

On 23/05/12 12:06, Christian Hoffmann wrote:

Unfortunately I can't provide a solution either -- I can just leave a "mee too" here and share some experience. I had the same problem: memory usage just grew over time. During testing, I also tried sending the data to /dev/null which did work for me. So my conclusion was that the encoder was too slow (for whatever reason -- it wasn't maxing out the CPUs). If sending to /dev/null exhibits the same problem for you, it might either be a different issue or a performance problem as well (but even without an encoder).

If the problem itself (encoding is delayed somewhere) cannot be solved, maybe it would be useful to drop frames in bmdcapture. I made some attempts to do so, but failed to drop video and audio frames properly, which led to problems in the encoder (waiting for the related audio frame which would never be sent because it was dropped...).

I have no idea how easy or hard it would be to implement that.

I'll try to implement something more graceful but IF the problem is slow processing there isn't much you can do that won't spoil the result.

I don't know which libav version you are using, but it might be possible, that some dev version leaked memory (didn't check), so using a release version might be a good idea.

libav is testrun on valgrind in our regression checking system, so it is unlikely.

(In my setup I just had to do live streaming for a specific time frame; just upping the bmdcapture memory limit worked for me because of that)

lu

Luca Barbato Gentoo/linux http://dev.gentoo.org/~lu_zero

[thenexxuz-zz]

The cards being used are the DeckLink SDI and the DeckLink Duo. I am using the following versions: libav-0.8.2-2.fc16.x86_64 bmdcapture (latest from github)

I don't think that the machine is having a problem processing frames because it is a server class box with a quad core and 6GB of ram and 1TB of HD space.

[thenexxuz-zz]

top - 09:18:13 up 2 days, 17:48,  3 users,  load average: 0.05, 0.06, 0.05
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  0.7%sy,  0.0%ni, 98.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   6103252k total,  2177248k used,  3926004k free,    15240k buffers
Swap:        0k total,        0k used,        0k free,   133288k cached

PID   USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17688 root      20   0 1892m 1.6g 3244 S 17.0 27.4 210:20.79 bmdcapture

This is bmdcapture running for about 20 hours with the following command:

/usr/bin/bmdcapture -C 0 -m 12 -A 2 -V 4 -F nut - -f pipe:1 > /dev/null

It appears that the memory issue is still there.

[lu-zero]

On 24/05/12 07:21, thenexxuz wrote:

top - 09:18:13 up 2 days, 17:48, 3 users, load average: 0.05, 0.06, 0.05 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 0.7%us, 0.7%sy, 0.0%ni, 98.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 6103252k total, 2177248k used, 3926004k free, 15240k buffers Swap: 0k total, 0k used, 0k free, 133288k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17688 root 20 0 1892m 1.6g 3244 S 17.0 27.4 210:20.79 bmdcapture

This is bmdcapture running for about 20 hours with the following command: /usr/bin/bmdcapture -C 0 -m 12 -A 2 -V 4 -F nut - -f pipe:1 > /dev/null

It appears that the memory issue is still there.

I'd need a valgrind run, looks like there is a memory leak.

lu

Luca Barbato Gentoo/linux http://dev.gentoo.org/~lu_zero

[thenexxuz-zz]

I used the following command:

valgrind --tool=memcheck --leak-check=full --show-reachable=yes --track-origins=yes -v /usr/bin/bmdcapture -C 0 -m 12 -A 2 -V 4 -F nut - -f pipe:1 > /dev/null

The output is listed here http://pastebin.com/u1kCbAH8

[digitalxzz]

I have the same problem with latest bmdcapture, libav 9_beta1 and BlackMagic SDK961. Still not exist solution for this problem ?

[thenexxuz-zz]

Coworkers have a workaround... cron restarts the capture process after about 9 to 13 hours. We have safeguards in place in case if it wants to restart during a scheduled time then it waits... we do lose about 3 minutes for the restart.

[thegrumpylion]

Any news regarding this issue?

[lu-zero]

I'm trying my best to track possible memory leaks and plug what valgrind reports. I also integrated in capture support in libav (check my local bmd branch and libbmd) to compare two different implementations of the same concept.

I'm running some tests as this week to compare the result.

Could you please report me exactly what you are doing when the problem appears?

if you just write to /dev/null what happens?

[ralawa]

I have the same issue. I compiled bmdtool with libavformat from avconv and ffmpeg. It is the same, the memory usage increases continuously. It is then impossible to use bmdtool for 24/7.

After 2 hours, 31488 root 20 0 425M 285M 2672 S 2.0 28.8 2:30.87 ./bmdcapture -m 11 -A 2 -V 4 -F nut -f pipe:1 The memory usage was 120M at startup and is 285M now.

[lu-zero]

ffmpeg is not supported. If you are capturing use my bmd branch from libav tree.

On Fri, Apr 19, 2013 at 11:29 AM, ralawa notifications@github.com wrote:

I have the same issue. I compiled bmdtool with libavformat from avconv and ffmpeg. It is the same, the memory usage increases continuously. It is then impossible to use bmdtool for 24/7.

After 2 hours, 31488 root 20 0 425M 285M 2672 S 2.0 28.8 2:30.87 ./bmdcapture -m 11 -A 2 -V 4 -F nut -f pipe:1 The memory usage was 120M at startup and is 285M now.

— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-16643973 .

[ralawa]

Where can we found your "bmd branch from libav tree"?

Thanks.

[lu-zero]

https://github.com/lu-zero/libav/tree/bmd

On Fri, Apr 19, 2013 at 2:03 PM, ralawa notifications@github.com wrote:

Where can we found your "bmd branch from libav tree"?

Thanks.

— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-16649402 .

[digitalxzz]

nut is the problem, for me works avi instead nut without memory leaking

2013/4/19 Luca Barbato notifications@github.com

https://github.com/lu-zero/libav/tree/bmd

On Fri, Apr 19, 2013 at 2:03 PM, ralawa notifications@github.com wrote:

Where can we found your "bmd branch from libav tree"?

Thanks.

— Reply to this email directly or view it on GitHub< https://github.com/lu-zero/bmdtools/issues/5#issuecomment-16649402> .

— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-16652133 .

[lu-zero]

On Fri, Apr 19, 2013 at 3:21 PM, digitalx notifications@github.com wrote:

nut is the problem, for me works avi instead nut without memory leaking

I'll triplecheck that, thanks for the pointer.

[ralawa]

Yes, bmdcapture works fine with avi. I have now an audio/video desync but I currently don't now if it comes from bmdcapture or something else.

[lu-zero]

On Tue, Apr 23, 2013 at 3:17 PM, ralawa notifications@github.com wrote:

Yes, bmdcapture works fine with avi. I have now an audio/video desync but I currently don't now if it comes from bmdcapture or something else.

Avi is horrible, I'll try to fix nut later if the problem is there... may I ask some volunteers for testing some variants soon?

[ralawa]

Yes, I'm volonteer.

[felixdreinulldrei]

I can confirm ralawas test results . With nut memory usage is increasing. With avi memory usage is low and stable but audio-/videosync issue starts. I would also test new variants.

[lu-zero]

Could you please use massif (part of the valgrind tool collection) and provide the ms_print results?

[felixdreinulldrei]

I've started valgrind. How many hours should I run the tool?

[lu-zero]

Once it starts filling up On Apr 28, 2013 5:21 PM, "felixdreinulldrei" notifications@github.com wrote:

I've started valgrind. How many hours should I run the tool?

— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-17136694 .

[felixdreinulldrei]

here is the link to my result. https://www.dropbox.com/s/yst4gcs2nkgbeuk/massif.out.1534

[felixdreinulldrei]

and here the result of a 12 hour capture process. https://www.dropbox.com/s/j8dqmqljzycc0xb/massif.out.1785

[CoMMyz]

Hi guys - i seem to have the same issue with memory usage.

Using latest Ubuntu 12.04 LTS with latest libav compiled from git and latest Blackmagic driver+sdk (9.7.1) with Blackmagic Intensity Pro.

Memory usage is growing - after about 28hrs process bmdcapture dies with Out of memory and the following syslog message.

http://pastebin.com/pEEW72b5

Machine has 4GB ram and its an i5 processor

[CoMMyz]

Hi,

If you use "asf" instead of "nut" or "avi" it works fine without a/v sync after 12hrs now.

-- Now 48hrs

[lu-zero]

Found out why using NUT causes the problem, I'll fix that problem soon on Libav. (on why I never noticed with valgrind memcheck, well, it is not a memory leak but a different usage assumption so the whole file index is being built in memory and never freed till the file isn't closed)

[amnonbb]

Is there any workaround to avoid problem with memory. I got same problem on bmdplay.. a memory is full within 24+ hours

[CoMMyz]

just use "asf" instead of "nut" :)

[amnonbb]

I tried.. :) in case bmdplay it's does not work

[lu-zero]

try to use the pipe mode in my nut branch.

On Tue, May 21, 2013 at 8:00 AM, AmnoNBB notifications@github.com wrote:

I tried.. :) in case bmdplay it's does not work

— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-18190686 .

[amnonbb]

Is that example of pipe mode? avconv ....... -f nut - | bmdplay -m 2 -f pipe:0 or maybe i don't understand something..

[lu-zero]

I crafted an experimental mode in nut so it doesn't keep an index and called that mode pipe (since you should use it only in that case) see the documentation in my libav nut branch.

On Wed, May 22, 2013 at 11:43 PM, AmnoNBB notifications@github.com wrote:

Is that example of pipe mode? avconv ....... -f nut - | bmdplay -m 2 -f pipe:0 or maybe i don't understand something..

— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-18310003 .

[amnonbb]

I have compiled from experimental branch but, how exactly to enter this format? i try: avconv ....... -f nut -nut_flags pipe - | bmdplay -m 2 -f pipe:0 and it gives me:

Unrecognized option 'nut_flags'. Error splitting the argument list:

how to format the line?

thanks

[felixdreinulldrei]

any idea why this happens sometimes? used command: bmdcapture -m 2 -V 1 -A 1 -F nut -f pipe:1|cat > /dev/null . it happens right after pressing enter. sometimes it does not happen.

[nut @ 0xf52260] pts < dts in stream 0 [nut @ 0xf52260] pts < dts in stream 1 [nut @ 0xf52260] pts < dts in stream 0 [nut @ 0xf52260] pts < dts in stream 1 [nut @ 0xf52260] pts < dts in stream 0 [nut @ 0xf52260] pts < dts in stream 1 [nut @ 0xf52260] pts < dts in stream 0 [nut @ 0xf52260] pts < dts in stream 1 [nut @ 0xf52260] pts < dts in stream 0

or

[asf @ 0x25ac260] pts < dts in stream 0 [asf @ 0x25ac260] pts < dts in stream 1 [asf @ 0x25ac260] pts < dts in stream 0 [asf @ 0x25ac260] pts < dts in stream 1 [asf @ 0x25ac260] pts < dts in stream 0 [asf @ 0x25ac260] pts < dts in stream 1

[felixdreinulldrei]

is there a way to make bmdcapture process kill itself and restart if any error like this occurs?

[lu-zero]

On Sat, May 25, 2013 at 4:03 PM, felixdreinulldrei <notifications@github.com

wrote:

is there a way to make bmdcapture process kill itself and restart if any error like this occurs?

I guess I should either open an irc channel or some other media, the bug tracker isn't the best place for this.

lu

[amnonbb]

Please somebody can explane how to use experimental pipe mode in libav nut branch

[CoMMyz]

I have tried your experimental avconv libav nut branch and successfully compiled it. But the problem with nut still persists.

machine kills process due to "oom-killer" as memory is exhausted completely.

[lu-zero]

please provide the exact command line and make sure the problem isn't due memory usage.

[m0zziter]

I'm experiencing this as well and I've tried using both the bmd and nut branches with the same result. Here's the command I'm running:

bmdcapture -C 1 -m 9 -V 4 -M 4 -F nut -f pipe:1 | avconv -y -i - -acodec aac -b:a 128k -strict experimental -vcodec libx264 -filter:v yadif -r 30 -b:v 900k -preset superfast -threads 4 -bt 2500 -qcomp 0.7 -f flv rtmp://myserver.com/

[jii2]

Hi

I think there is some problem with memory handling. I have DeckLink's Quatro card with three inputs enabled and after a while inputs randomly will give me an error:

root@ODE:/DVB# bmdcapture -m 2 -C 2 -A 2 -V 4 -F asf -f pipe:1 Failed to enable video input. Is another application using the card? Segmentation fault (core dumped)

I'm using this card 24/7 and start and stop automatic capture hundreds of times per a day. When I get this error, only way to make input to work again is reboot.

So my question is, that how I can avoid this and/or is there something that can do with orginal code to fix this?

ps. I'm using latest ubuntu server/kernel, latest bmdtools and latest drivers from declink.

Br, Jiii

[lu-zero]

On Jul 25, 2013 8:29 AM, "jii2" notifications@github.com wrote:

Hi

I think there is some problem with memory handling. I have DeckLink's Quatro card with three inputs enabled and after a while inputs randomly will give me an error:

root@ODE:/DVB# bmdcapture -m 2 -C 2 -A 2 -V 4 -F asf -f pipe:1 Failed to enable video input. Is another application using the card? Segmentation fault (core dumped)

A backtrace might be nice even if I doubt I can fix the problem.

I'm using this card 24/7 and start and stop automatic capture hundreds of times per a day. When I get this error, only way to make input to work again is reboot.

So my question is, that how I can avoid this and/or is there something that can do with orginal code to fix this?

Ask blackmagic for driver support. If you do not have zombie bmdcapture...

[amnonbb]

I got 17140 Segmentation fault | ~/bmdplay -C 1 -b 0 -f pipe:0 when tried experimental pipe with bmd branch NUT without flags work but still with memory problem

[felix303]

any new experiences with memory leak problem? maybe some solution meanwhile? using the latest drivers Desktop Video 10.2.3, I am still experiencing increasing memory.

[digitalxzz]

Hi,

this is problem only with NUT. When you use different container, everything is OK. For me work very good stupid .avi container.

Majo

2014-10-28 19:17 GMT+01:00 felix303 notifications@github.com:

any new experiences with memory leak problem? maybe some solution meanwhile? using the latest drivers Desktop Video 10.2.3, I am still experiencing increasing memory.

— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-60804607.