Docker Container Crashes with "Could not find track info for track ID 2" Error for Some RTSP Streams

[dongeorge77]

Description

I am using the OvenMediaEngine within a Docker container, and I've encountered a problem when creating multiple RTSP streams using the API (nearly 20 RTP streams). Specifically, when I try to play some of these streams, I receive a "Could not find track info for track ID 2" error, and eventually, the Docker container crashes.

Steps to Reproduce

1. Start OvenMediaEngine in a Docker container.
2. Use the API to create multiple RTSP streams with different names and URLs.
3. Attempt to play some of these streams.

Expected Behavior

1. All RTSP streams should play without errors, as they can be streamed successfully using VLC media player.

Actual Behavior

1. Some RTSP streams generate a "Could not find track info for track ID 2" error, leading to the Docker container crashing.

Please let me know if there is any additional information or logs required to investigate and resolve this issue.

[getroot]

Thanks for the report.

1. Which version of OvenMediaEngine are you using?
2. Full logs are helpful for analysis.
3. Please provide the RTSP URL to help with analysis. (If you cannot disclose it, please send it to support@airensoft.com.)

[dongeorge77]

OvenMediaEngine version: 0.15.16

Sorry @getroot , I can not provide a complete log file since it contains potential RTSP URLs.

Attaching some of the log screenshots below,

[getroot]

I cannot reproduce this issue, so if you could provide the RTSP URL, we can help resolve the problem. Would you be able to provide your RTSP URL to support@airensoft.com?

Additionally, there should be a crash log file located in the directory where OvenMediaEngine binary is (/opt/ovenmediaengine/bin). Providing that would be helpful.

[getroot]

Is it impossible to provide an RTSP URL?

The message "Could not find track info for track ID 2" is not a clue. This is a log that occurs because the RTSP server is not sending data on the initially agreed channel, and it is not the cause of the crash. Furthermore, this log is unrelated to the publisher (playback) and is specific to the RTSP input. Therefore, I cannot analyze this issue further based on the logs you provided.

It is suspected that the crash is occurring elsewhere, and to analyze it, I need access to your RTSP server. Therefore, I require your RTSP access permissions.

Also, could you please specify whether you are attempting to play using WebRTC or LLHLS?

[dongeorge77]

During the time when the server crashed, there was no WebRTC streaming taking place.

please note that, The server didn't crash because of a single stream failure; it crashed due to multiple stream failures. This led to a situation where the container became chocked and eventually crashed. I observed this by using htop within the container, and I noticed a significant number of processes being created, which likely contributed to the issue.

[the-real-rusty-shackleford]

I have seen this happen for IP cameras that have audio enabled, but that are using the G.711 codec, which is a common codec for IP cameras. This codec is not currently supported by OvenMediaEngine.

For some camera brands, I was able to disable audio or change the codec to AAC and the errors went away. For others, I could not disable audio or change the codec and so my logs are flooded with that error message every few milliseconds... it would be nice if there were a way to prevent that. Perhaps an option to ignore audio tracks for an incoming stream. I saw there was a PR for this (#1246) but it was rejected. Or maybe OME could detect the codec and if it's unsupported, it could ignore the track.

@dongeorge77 Which audio codecs do your streams use?

[getroot]

@the-real-rusty-shackleford Thank you for providing good information. The ability to reject unsupported codecs is already included in the code, but if it doesn't work, it's a bug. I will check it out.

1246 is a function that disables the entire audio, and I will check whether the function that rejects unsupported codec is working properly.

[the-real-rusty-shackleford]

A minimum example for the codec issue is below.

Files

Server.xml

<?xml version="1.0" encoding="UTF-8" ?>

<Server version="8">
    <Name>OvenMediaEngine</Name>
    <!-- Host type (origin/edge) -->
    <Type>origin</Type>
    <!-- Specify IP address to bind (* means all IPs) -->
    <IP>*</IP>
    <PrivacyProtection>false</PrivacyProtection>
    <!-- 
    To get the public IP address(mapped address of stun) of the local server. 
    This is useful when OME cannot obtain a public IP from an interface, such as AWS or docker environment. 
    If this is successful, you can use ${PublicIP} in your settings.
    -->
    <StunServer>stun.ovenmediaengine.com:13478</StunServer>

    <Modules>
        <LLHLS>
            <Enable>true</Enable>
        </LLHLS>
    </Modules>

    <!-- Settings for the ports to bind -->
    <Bind>
        <!-- Enable this configuration if you want to use API Server -->
        <Managers>
            <API>
                <Port>8081</Port>
                <WorkerCount>1</WorkerCount>
            </API>
        </Managers>
        <Providers>
            <!-- Pull providers -->
            <RTSPC>
                <WorkerCount>1</WorkerCount>
            </RTSPC>
            <OVT>
                <WorkerCount>1</WorkerCount>
            </OVT>
            <!-- Push providers -->
            <RTMP>
                <Port>1935</Port>
                <WorkerCount>1</WorkerCount>
            </RTMP>
        </Providers>

        <Publishers>
            <LLHLS>
                <Port>3333</Port>
                <WorkerCount>1</WorkerCount>
            </LLHLS>
        </Publishers>
    </Bind>
    <Managers>
        <Host>
            <Names>
                <Name>*</Name>
            </Names>
        </Host>
        <API>
            <AccessToken>test:test</AccessToken>
            <CrossDomains>
                <Url>localhost</Url>
            </CrossDomains>
        </API>
    </Managers>
    <VirtualHosts>
        <VirtualHost>
            <Name>default</Name>

            <Host>
                <Names>
                    <Name>*</Name>
                </Names>
            </Host>

            <!-- Settings for applications -->
            <Applications>
                <Application>
                    <Name>app</Name>
                    <!-- Application type (live/vod) -->
                    <Type>live</Type>
                    <OutputProfiles>
                        <OutputProfile>
                            <Name>bypass_stream</Name>
                            <OutputStreamName>${OriginStreamName}</OutputStreamName>

                            <Encodes>
                                <Video>
                                    <Name>bypass_video</Name>
                                    <Bypass>true</Bypass>
                                </Video>
                                <Audio>
                                    <Name>bypass_audio</Name>
                                    <Bypass>true</Bypass>
                                </Audio>
                            </Encodes>
                        </OutputProfile>
                    </OutputProfiles>
                    <Providers>
                        <RTMP />
                        <RTSPPull />
                    </Providers>
                    <Publishers>
                        <AppWorkerCount>1</AppWorkerCount>
                        <StreamWorkerCount>1</StreamWorkerCount>
                        <LLHLS>
                            <OriginMode>true</OriginMode>
                            <EnablePreloadHint>false</EnablePreloadHint>
                            <ChunkDuration>0.5</ChunkDuration>
                            <PartHoldBack>1.5</PartHoldBack>
                            <SegmentDuration>6</SegmentDuration>
                            <SegmentCount>10</SegmentCount>
                            <CrossDomains>
                                <Url>*</Url>
                            </CrossDomains>
                        </LLHLS>
                    </Publishers>
                </Application>
            </Applications>
        </VirtualHost>
    </VirtualHosts>
</Server>

data-pull.json

{
  "name": "test",
  "urls": ["rtsp://host.docker.internal:8554/test"]
}

run.sh

OME_TAG=${1:-0.16.3}

docker image pull airensoft/ovenmediaengine:$OME_TAG
docker image pull jrottenberg/ffmpeg:4.4-alpine
docker image pull bluenviron/mediamtx:latest

docker container rm ome-test --force
docker container rm ffmpeg-test --force
docker container rm rtsp-test --force

echo "Starting OME..."

docker run --rm -d \
  --name ome-test \
  -p 1935:1935 \
  -p 8081:8081 \
  -v ./conf:/opt/ovenmediaengine/bin/origin_conf \
  airensoft/ovenmediaengine:$OME_TAG

echo "Starting rtsp server..."

docker run --rm -d \
  --name rtsp-test \
  -p 8554:8554 \
  bluenviron/mediamtx:latest

echo "Starting ffmpeg..."
sleep 5

docker run --rm -d --name ffmpeg-test -v $(pwd):$(pwd) jrottenberg/ffmpeg:4.4-alpine \
  -re \
  -stream_loop \
  -1 \
  -i $(pwd)/test-with-audio.mp4 \
  -c:v libx264 \
  -c:a pcm_alaw \
  -preset fast \
  -f rtsp \
  -rtsp_transport tcp \
  rtsp://host.docker.internal:8554/test

echo "Trying to pull stream..."

while curl \
  -s \
  -X POST \
  -H "Content-Type: application/json" \
  -d @data-pull.json \
   http://test:test@localhost:8081/v1/vhosts/default/apps/app/streams | grep 502 > /dev/null
do
  sleep 1
done

echo "Stream pulling."

Reproduction Steps

Overview

This example uses an RTSP server running as a docker container, since RTMP does not support G.711 codecs. In this example we

1. Start OME
2. Start an RTSP server
3. Start ffmpeg in a container, reading in an mp4 file encoded as h264 + aac. FFMPEG transcodes the audio to G711 (PCM) and sends it to the RTSP server.
4. The OME API is invoked to start pulling the RTSP stream from the RTSP server.

Steps

1. Put Server.xml in ./conf/.
2. Put run.sh and data-pull.json into ./.
3. Put an MP4 file with audio into ./ and call it test-with-audio.mp4.
4. Run run.sh.
5. Run docker logs ome-test --follow and observe that the logs are flooded by [2023-11-30 17:18:19.133] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334 | Could not find track info for track ID 2.

Now try changing the audio codec to aac by updating the ffmpeg flag -c:a pcm_alaw to -c:a aac. Run the test again and see that the error message is gone.

The expected behavior is that if an audio codec is not supported, the logs should not be flooded with the error message. Below is a small sample of the logs.

[2023-11-30 17:18:19.123] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.124] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.128] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.132] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.133] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.135] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.153] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.154] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.156] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2
[2023-11-30 17:18:19.157] E [StreamMotor:21] RtpRtcp | rtp_rtcp.cpp:334  | Could not find track info for track ID 2

Because this error fires hundreds of times a second, the logs are difficult to navigate and they take up a lot of space.

Before the spamming starts, OME does at least recognize that the codec is invalid. See the log message below.

[2023-11-30 16:56:49.755] E [SPAPISvr-t8081:20] RtspcStream | rtspc_stream.cpp:541  | test - Unsupported codec  : 2

[getroot]

@the-real-rusty-shackleford This is a really great bug report. I will resolve this issue soon. thank you

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[the-real-rusty-shackleford]

Bump.

[getroot]

@the-real-rusty-shackleford I improved it so that unsupported codecs are not negotiated during the rtsp nego process. You can test it with the latest master branch. thank you.

[getroot]

This patch made it into the 0.16.5 release. If the issue reproduces, please reopen the issue. Thank you.