RTSP stream with video stops working and freezes on first frame when audio is added

[tobias-indico]

Version

Media3 1.2.0

The issue/bug details

I have set up a local RTSP stream with GStreamer, and are experiencing some problems when trying to stream the audio. What happens is that the stream wont play in ExoPlayer (it works in VLC using libVLC) when I try to stream the audio, this happens both when there is only audio, and if I play video+audio (only video works fine in ExoPlayer).

I have checked that the audio is AAC (using ffmpeg) in my mp4-file that I stream, but maybe GStreamer does something to it since it's not working in ExoPlayer. However if I add some extra parameters when setting up the stream (not just using raw video and audio) and adding some "parsing to aac" (in GStreamer python script) the RTSP stream works with both video and audio in ExoPlayer.

Im very new to streaming and might misunderstand what audio-types that is compatible with ExoPlayer and not, but as mentioned the stream which was sent in its simplest form worked in VLC and not in ExoPlayer, which I find odd since its a common formatted stream.

Some logging

Here is some logging from Android Studio, the first one is for the not working stream and the second is for the working stream (Note that the parameter "bitrate" is present only in the working stream):

Stream that doesn't work tracks [eventTime=0.37, mediaPos=0.00, window=0, period=0 group [ [X] Track:0, id=null, mimeType=video/avc, codecs=avc1.4d401e, res=640x360, color=BT709/Limited range/SDR SMPTE 170M/8/8, supported=YES ] group [ [X] Track:0, id=null, mimeType=audio/mp4a-latm, codecs=mp4a.40.30, channels=2, sample_rate=44100, supported=YES ] ]

Stream that works tracks [eventTime=0.47, mediaPos=0.00, window=0, period=0 group [ [X] Track:0, id=null, mimeType=video/avc, codecs=avc1.4d401e, res=640x360, color=BT709/Limited range/SDR SMPTE 170M/8/8, supported=YES ] group [ [X] Track:0, id=null, mimeType=audio/mp4a-latm, bitrate=128000, codecs=mp4a.40.2, channels=2, sample_rate=44100, supported=YES ] ]

Devices that reproduce the issue

Samsung Galaxy S22 running Android 14 Pixel 2 emulator running Android 11

Devices that do not reproduce the issue

None (so far)

Reproducible in the demo app?

Not tested (couldn't find test for RTSP stream)

Reproduction steps

For me I had to set up a RTSP stream of a video file that had AAC audio (I used a mp4 video file), and that would not be playable in ExoPlayer. I used GStreamer and/or VLC on Mac to set up the local streams.

Expected result

The video should play with audio (synced to the video)

Actual result

The first frame of the video is shown, and the max duration is inserted into the "duration" element in the ExoPlayer controller, but the current duration is stuck at 0 (not playing).

Media

https://github.com/androidx/media/assets/60511356/67060b6d-0289-4a83-ab31-731506ef3f87

[tobias-indico]

UPDATE:

A recent discovery I made is that the "non playable" RTSP stream actually will play if the whole video is done buffering. The video that is playable is without sound.

I'm not sure if it is still a bug, or if it is now just some settings I need to change.

An important detail is that I see in the logging that the playback state will be set to "ready", but at the same instant (some milliseconds after) it will reset and go back to playback state "buffering". So the signal "ready" is never being sent and stays until a whole RTSP stream video is buffered.

Any ideas how to bypass that playback state "buffering" overrides "ready" immediately (until the whole video is buffered)?

[tobias-indico]

I finally found a solution. It works fine if we add a decoder extension (I used ffmpeg) and then add it to ExoPlayer.

IMPORTANT: In addition to adding extension decoder we must use "EXTENSION_RENDERER_MODE_PREFER" as it will not work with "EXTENSION_RENDERER_MODE_ON" because "MediaCodecAudioRenderer" will otherwise be used since it registerts the audio as "supported".

There is clearly something weird (a bug) going on here since what ExoPlayer thinks is supported is not actually working and/or supported.

[wman1980]

Hi, I am having the same problem see https://github.com/androidx/media/issues/1401.

Any advice?

Thanks

[tobias-indico]

Hello, my main solution was, unfortunately, to switch to the "VLC for Android" library.

However, you might be able to make it work with ExoPlayer by adding "FFMPEG" as a decoder module. If I remember correctly, this worked for me, but I ended up using the "VLC for Android" library anyway since it worked better.

The guide to set up the FFMPEG Decoder Module for ExoPlayer is here: FFMPEG Decoder Module. When following that guide, it is important to choose "aac" when setting ENABLED_DECODERS.

I hope this helps. However, if you are like me, then temporarily using "libVLC" while hoping/waiting for this to be fixed/added in Media3/ExoPlayer might be something to consider, since it could save you some time and work :)

[wman1980]

Hi Tobias,

I have setup the FFMPEG Decoder Module as described. Building via ./build_ffmpeg.sh works. Unfortunately I still can not stream my rtsp stream with audio enabled.

In my activity I setup exoplayer as follows:

val defaultRenderersFactory = DefaultRenderersFactory(this).setExtensionRendererMode( DefaultRenderersFactory.EXTENSION_RENDERER_MODE_PREFER ) val player = ExoPlayer.Builder(this).setRenderersFactory(defaultRenderersFactory).build() val playerView = findViewById(R.id.player_view) playerView.player = player val mediaItem = MediaItem.fromUri("rtsp://user:pwd@192.168.4.62:554/livestream/13") player.setMediaItem(mediaItem) player.addAnalyticsListener(EventLogger()) player.prepare() player.play()

My dependencies are:

implementation("androidx.media3:media3-exoplayer:1.3.1") implementation("androidx.media3:media3-exoplayer-dash:1.3.1") implementation("androidx.media3:media3-ui:1.3.1") implementation("androidx.media3:media3-exoplayer-rtsp:1.3.1")

Do I need to include the 'media-lib-decoder-ffmeg' module to my dependencies? E.g. something like this:

implementation(project("media-lib-decoder-ffmpeg"))

It's playing if audio is disabled. Not sure whats still going wrong ...

[wman1980]

ahhhh, fixed it by changing the following things:

settings.gradle.kts:

enableFeaturePreview("TYPESAFE_PROJECT_ACCESSORS")

build.gradle.kts:

compileOptions { isCoreLibraryDesugaringEnabled = true sourceCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_1_8 } dependencies { implementation(projects.mediaLibExoplayer) implementation(projects.mediaLibUi) implementation(projects.mediaLibExoplayerRtsp) implementation(projects.mediaLibDecoderFfmpeg) coreLibraryDesugaring("com.android.tools:desugar_jdk_libs:2.0.3") }

Now my RTSP stream is playing with audio!

You can check if exoplayer loaded the FfmpegAudioRenderer by checking the troubleshooting page:

if you're using DefaultRenderersFactory, you should see an info-level log line like "Loaded FfmpegAudioRenderer" in Logcat when the decoder loads. If that's missing, make sure the application has a dependency on the decoding library.

Awesome & thanks!

[tobias-indico]

Great! In my case, I didn't need to implement "dash," but if you are using DASH in your project, then everything looks good :)