Low-latency RTSP or UDP playback

[rayw-dronesense]

The environment is a MAVLink controller (based on Android 10) and Drone setup, the video stream arrives on a local UDP port. Using ExoPlayer part of media3 v1.2.1.

    private val videoRenderer = object : MediaCodecVideoRenderer(context,
        MediaCodecSelector { mimeType, requiresSecureDecoder, requiresTunnelingDecoder ->
            MediaCodecSelector.DEFAULT
                .getDecoderInfos(mimeType, requiresSecureDecoder, requiresTunnelingDecoder)
                .filterNot { it.hardwareAccelerated }
                .toMutableList()
        }) {}

...
            val udpUri = "udp://0.0.0.0:${udpPort}"
            val dsf = DataSource.Factory { UdpDataSource() }
            val mediaitem = MediaItem.Builder()
                .setUri(Uri.parse(udpUri))
                .setLiveConfiguration(
                    MediaItem.LiveConfiguration.Builder().setMaxPlaybackSpeed(1.02f).build()
                )
                .build()
            val mediaSource = ProgressiveMediaSource.Factory(dsf).createMediaSource(mediaitem)
            // Force software rendering to ensure frames arrive in standard format.
            val renderersFactory = RenderersFactory { _,_,_,_,_ ->
                arrayOf(videoRenderer)
            }
            // Disable buffering behavior, we want the fastest possible updated stream.
            val loadControl = DefaultLoadControl.Builder()
                            .setBufferDurationsMs(0, 500, 0, 0)
                            .setPrioritizeTimeOverSizeThresholds(true)
                            .build()
                val player = ExoPlayer.Builder(context)
                    .setRenderersFactory(renderersFactory)
                    .setLoadControl(loadControl)
                    .build()
                player.setMediaSource(mediaSource)
                player.setVideoSurface(imageReaderHelper.surface)
                player.prepare()

We are seeing 2-3 seconds delay between what actually happens and what we see rendered by ExoPlayer in the ImageReader.

We do not mind if ExoPlayer drops frames to achieve lower latency.

Could you please help us with this matter? Thanks!

[tonihei]

This question came up before, for example in https://github.com/google/ExoPlayer/issues/11221 or https://github.com/google/ExoPlayer/issues/10694, most often combined with RTSP playback. The short answer is that you try to set some config values like the buffer durations to something really small, but otherwise the player isn't really built with this use case in mind.

Given we now had many different requests in the same direction, I'm going to mark this one as a feature request for a "low-latency" mode through the player that would allow to stream these time-sensitive local network input streams with the smallest possible delay. We don't get around to create such a mode soon though, so also marking as "low priority" for now.

[markg85]

I might need to make a separate issue for this but i thought i'd mention it here first.

A low latency mode that would just dump the frames on screen as they are decoded would be awesome to have!

A use case i have for this is desktop streaming (same case for game streaming). I for example want to use samsung dex (or the android desktop mode) which requires:

• A low latency decode step (we have that since android 11 provided that the soc vendors made low latency codec versions)
• A transport mechanism. That would be TCP and UDP, media 3 supports those.
• And a "wire format" that allows the lowest latency possible. This probably is mpegts in terms of what media3 supports as "extractors".

I'm curious if it would make sense for the devs of media3 to support the ffmpeg nut format (see the code). I keep falling back on that container format if i want to have the lowest possible latency. Note that i currently use that very format in a pc to pc streaming manner within a local network.

Having a pure video stream format (say h.264, h.265, AV1, ...) isn't the best as you'd be on your own to detect where one frame starts and the next one begins. Hence a simple container - like NUT - makes this a lot simpler at effectively no added latency cost.

[alexeyvasilyev]

You can check this library designed specifically for low latency RTSP stream https://github.com/alexeyvasilyev/rtsp-client-android