Bug: ffmpeg consumer constant failure - cannot allocate memory

paveltonev commented 4 months ago

Observed Behavior

Use CasparCG server v2.4.0-rc1 with ffmpeg 6.1.1 and start ffmpeg consumer started via:

<consumers>
    <ffmpeg>
        <path>udp://127.0.0.1:9094?pkt_size=1316&bitrate=8000000&reuse=1&overrun_nonfatal=1&max_interleave_delta=1000000&fifo_size=xxxxxxx&buffer_size=yyyyyyyyy</path>
        <args>-codec:v h264_nvenc -preset:v p7 -profile:v high -tune:v ll -level:v 4.2 -g:v 50 -rc:v cbr -b_ref_mode:v middle -temporal-aq:v 1 -rc-lookahead:v 20 -i_qfactor:v 0.75 -b_qfactor:v 1.1 -bufsize:v 0.7M -max_delay 0.7M -pcr_period 30 -minrate:v 7M -maxrate:v 7M -b:v 7M -bf:v 2 -qmin:v 0 -ac:a 2 -channel_layout:a stereo -codec:a aac -ar:a 48000 -b:a 256K -bufsize:a 128K -minrate:a 256K -maxrate:a 256K -format mpegts -muxrate 8M -mpegts_flags +system_b -fflags +genpts+nofillin -flush_packets 0 -mpegts_pmt_start_pid 0x1500 -mpegts_start_pid 0x150 -mpegts_service_type advanced_codec_digital_hdtv</args>
    </ffmpeg>
</consumers>

with CasparCG client v2.3.0-rc1 executing a 48 hours playlist. Everything started as expected and on the 12th minute the ffmpeg cosumer terminates with Cannot allocate memory issue described here. It appeared that this error has already been reported as issue on ffmpeg trac system here.

I increased 10 times the default fifo_size and buffer_size values, as a result the ffmpeg consumer failed in around 120 minutes.

I increased 20 times the default fifo_size and buffer_size values, as a result the ffmpeg consumer failed in around 240 minutes.

Expected behaviour

I expect ffmpeg consumer not to throw such an error.

Steps to reproduce

Install CasparCG server v2.4.0-rc1 under Ubuntu 22.04
Create consumer with the above configuration without fifo_size and buffer_size settings.
Use CasparCG client v2.3.0-rc1 under Ubuntu 22.04 and create a more than 12 hours playlist
Wait for no more than 20 minutes, the consumer will fail with the issue described above.
Increase fifo_size and buffer_size proportionally to their default values, the failure is proportional to the change.

Environment

Commit: https://github.com/CasparCG/server/releases/tag/v2.4.0-rc1
Server version: Linux mint with Ubuntu 22.04
Operating system: Ubuntu 22.04 with Kernels 5.15, 6.27

Julusian commented 4 months ago

I'm struggling to reproduce this, the command provided above does not work with ffmpeg 4.4, which is what ubuntu 22.04 uses by default.

paveltonev commented 4 months ago

I use ffmpeg 6.1.1 built from source with NVIDIA GPU. The workflow is following:

Install NVIDIA driver
Install CUDA from https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#ubuntu
Build from source FFMPEG with all required modules and CasparCG. Use this file install_direct_6.1.1.sh.zip as a reference and change INSTALL_PATH to point your project location.

paveltonev commented 4 months ago

I did a test also with v2.4.0-stable from yesterday and same error appeared. Please find attached the log, where I started 5 channels with no fifo_size and buffer_size and 4 of them terminated with "Cannot allocate memory" error. nohup.out.zip

jpc0 commented 3 months ago

I just ran the same output with a random input video directly in ffmpeg and got the same issue.

From the final comment in the trac thread you posted

[comment:26](https://trac.ffmpeg.org/ticket/6463#comment:26) by Marton Balint, [5 years ago](https://trac.ffmpeg.org/timeline?from=2019-03-17T00%3A10%3A58%2B02%3A00&precision=second)

It is most probably not a memory issue, simply there is not enough space in the output fifo. Error message is a bit misleading. Try increasing the fifo size on the output.

Does not seem to be an issue with casparcg? Could you try the same test.

CasparCG / server