Live Media with RunnigText in the bottom - text is moving from side to side

[giladna]

Hi,

We have live media where there is running text from right to left at the bottom of the video

While using this media in Exo Player on Android TV box(especially) we can see video looks ok but the running text is moving from side to side (shaking - that makes you dizzy )or getting stuck for fraction of second

Our media experts says that it is ingested correctly and the media is just fine

I am not sure where issue is and would much appreciate if you could point me for the root cause of this issue. if is it decoders limitations or frames skips or live packets sync or frame rate or if there are any steps Exo Player API can help to make the text running stable and constantly running in fluent manner

this media has 1 50fps(5.4Mbps) track and 25fps tracks in lower bitrates

While playing the media I can see in the log cat errors regarding the decoders that are being used I tried the other decoder in the device instead the awsome2 decoder but I could only see it got worst.

-07 13:39:26.299  4412  9367 E ACodec  : [OMX.amlogic.avc.decoder.awesome2.secure] setPortMode on output to DynamicANWBuffer failed w/ err -2147483648
12-07 13:39:26.304  4412  9367 I ACodec  : codec does not support config priority (err -1010)
12-07 13:39:26.305  4412  9367 I ACodec  : codec does not support config operating rate (err -1010)
12-07 13:40:02.921  4412  9367 D SurfaceUtils: connecting to surface 0xb475e808, reason pushBlankBuffersToNativeWindow(err2)
12-07 13:40:02.992  4412  9468 E ACodec  : [OMX.amlogic.avc.decoder.awesome2.secure] setPortMode on output to DynamicANWBuffer failed w/ err -2147483648
12-07 13:40:02.996  4412  9468 I ACodec  : codec does not support config priority (err -1010)
12-07 13:40:02.996  4412  9468 I ACodec  : codec does not support config operating rate (err -1010)

Versions:

Exo Player v2.15.1 Android TV box - Android (ZTE android 10, NVIDIA SHEILD Android 9, Xiomi mii box Android 9)

I am sending in mail the files that we use (ts format) I am sending the BugReport as well

please notify me how I can help if any issue with reproduction.

10x

GIlad

[icbaker]

I played the provided TS file using the ExoPlayer demo app on a Pixel 5 and a Chromecast with Google TV. On both devices the text didn't seem to be shaking significantly. On the larger screen of the Chromecast I did see some compression artefacts in the video just above the moving text - I suspect this is due to the way the video is encoded and the poor interaction between scrolling text and conventional video compression techniques.

I don't think there's anything actionable from ExoPlayer here. I suspect this is related to the way the video is encoded: the choice of codec and codec parameters.

[icbaker]

On the larger screen of the Chromecast I did see some compression artefacts in the video just above the moving text

I also see exactly the same compression artefacts when playing the video on VLC on MacOS (on the same screen).

[giladna]

Thanks @icbaker for looking at this

[OrenMe]

Do you see any meaningful errors from decoder that we can maybe try to take to the encoding provider to tweak their configuration?

[pwuamlogic]

Hi,

This is Paul WU from Amlogic looking at this issue from decoder side. Here are some findings from a log captured yesterday in lab.

1. We add a patch (attached) onto framework/av to dump all the rendering or dropping of frames: frameworks_av_MediaCodec_ACodec_render.zip

2. With it, we got this log (tmp6 as attached) when the reported flickering text happens: tmp6.zip

3. Looking into the keyword "onOutputBufferDrained" where all drops and render will be printed, we find that the render timestamp jumps frequently. For example, the 2 adjacent frames with PTS interval of 20ms (50fps), has a render time interval of 40ms.

We also got a stats of the whole log, and found out that jumping render time without normal 20ms happens frequently, about once per second. |logged at | PTS | render at | |12-13 15:34:23.880000| 1639406050720000| 149464917496| |12-13 15:34:24.559000| 1639406051380000| 150144917143| |12-13 15:34:25.231000| 1639406052020000| 150804913557| |12-13 15:34:25.874000| 1639406052660000| 151464912763| |12-13 15:34:26.554000| 1639406053320000| 152144913428| |12-13 15:34:27.219000| 1639406053960000| 152804914668| .... totally 368 records ...

4. Meanwhile, the PTS seems very stable: |logged at | PTS | render at | |12-13 15:34:40.461000| 1639406066820000| 166044914824| |12-13 15:37:14.982000| 1639406219580000| 320564914821| |12-13 15:40:57.962000| 1639406442560000| 543544916377| |12-13 15:43:38.366000| 1639406609800000| 703907432333| |12-13 15:43:42.427000| 1639406613120000| 707967489835| |12-13 15:43:46.564000| 1639406616260000| 712104793045| |12-13 15:43:47.485000| 1639406616460000| 713004910837| |12-13 15:46:10.641000| 1639406755340000| 856224912560| 8 records in total

So far, we think if the render time of frames is not stable, the running text will flicker. And AFAIK, render time is specified by Player via SDK interface: so we suggest to look into the player implementation to know how the render time is calculated in the case. Appreciate your information by advance.

Regards, Paul

[tonihei]

The release timestamps are adjusted to match with the display's vsync times. That's why it's jumping by a full additional 20ms instead of increasing gradually.

We are aware of some device-specific issues on some TVs where live speed adjustment causes visible dropped frame effects every 2 seconds. Could you try setting: MediaItem.Builder.setLiveMaxPlaybackSpeed(1f) in your MediaItem? Just to check if this is the same issue.

[giladna]

@tonihei @icbaker

Thanks, I will check this suggestion and update

10x Gilad

[giladna]

@tonihei Hi,

It seems that setLiveMaxPlaybackSpeed(1f) fixes this behaviour for live media thanks!

This live stream has catchup and this flag is not helping for this case since it is not live anymore how can we handle this behaviour in live streams that were recorded for future use? 10x

[tonihei]

Thanks for testing!

Could you let us know exactly on which devices you observed this to be a problem? So far we know of two TV types that have this problem (Nvidia Shield Pro 2019 Android 9 and a Sony Bravia 4K), other devices like ChromeCast and all phones we tested with don't exhibit the problem. We will disable speed adjustment automatically (independent of content) on devices known to have problems, but we need to identify the right scope first. Ideally, we'd like to know the device info strings (the log line starting with ExoPlayerImpl: Init or ExoPlayerImpl: Release) and the name of the decoder (log lines starting with videoDecoderInitialized or videoDecoderReleased).

This live stream has catchup and this flag is not helping for this case since it is not live anymore how can we handle this behaviour in live streams that were recorded for future use?

I'm not sure I fully understand what you mean. Are you saying that you see this issue in non-live content despite setting the max playback speed to 1f? This is surprising in multiple ways because non-live content should never use the speed adjustment to start with (if it does, it's a bug) and if you are setting this flag to 1f, it should set it everywhere (even if such a bug exists). Could you clarify for what type of content you are seeing this issue? If you can, you could also send us a link to the content to dev.exoplayer@gmail.com with "Issue #9773" in the subject, so we can check how the content is handled by ExoPlayer.

[giladna]

@tonihei I assumed that I should not set this flag if media is not live media (setLiveMaxPlaybackSpeed)

should I?

[tonihei]

It shouldn't make a difference at all, the value is only applied if the content is live.

[giladna]

@tonihei

OK, So i did not set it al all for none live media in my code but the issue is there since the recorded media has same pattern as live media the only thing is that we cannot influance on the behaviour

I am also wondering how I can be sure if the problem is on player side or media side as Ian wrote it was observed on VLC for him

[tonihei]

I think we might be talking about different issues here:

1. A video decoding artefact that is related to the media (and is visible in VLC as well).
2. A device-specific rendering problem related to speeding up the media during live playback.

Issue (2) should only affect live playbacks and if setting the max speed to 1f helps to solve it, it would be great if you could let us know which devices it affects so that we can add this workaround to the ExoPlayer library itself.

Issue (1) is not something we can fix on the player side because it's probably related to your encoding process and settings.

Does that fit to what you are seeing?

[giladna]

@tonihei yes this observation is correct though I have no clue how to provide proof that there is media issue there.

we have a concern regarding this fix, if we start play with the speed config issue resolved but, we could see that if we seek back in the live stream then issue is there again. could you tell why this happens or how to force it back to 1.0 after seek?

10x Gilad.

We could see that this live speed to 1.0 helps for rendering issue tested on:

• MediaBox_B866V2, MediaBox, ZTE, 29
• oneday, MIBOX4, Xiaomi, 28
• DV8519, SDMC1901, SDMC, 28
• mdarcy, SHIELD Android TV, NVIDIA, 28

I think that it is some how connected as well to the configured screen resolution, when I configure it to be 1080p 50 Hz it is better than the default 1080p 59.940Hz

[tonihei]

we have a concern regarding this fix, if we start play with the speed config issue resolved but, we could see that if we seek back in the live stream then issue is there again.

This sounds as if the underlying device issue comes up in more ways than just live speed adjustment. Setting the value to 1f once always keeps it that way. The fact that you are still seeing the issue after seeking and also for VOD streams suggests that this is generally happening on these devices.

I think that it is some how connected as well to the configured screen resolution, when I configure it to be 1080p 50 Hz it is better than the default 1080p 59.940Hz

Our initial investigation came to a similar conclusion that it depends on some internal device screen refresh logic that doesn't interact well with how we release the video frames. The released timestamps all looked good (and the same as they look on mobile phones), so there must be something off in how these frames are shown on the screen. We also got reports that setting the screen refresh rate manually to match the nominal media frame rate delivers the best results, but this is of course infeasible as you could hardly ask your users to change their TV setting before using your app.

[giladna]

@tonihei

I came across this guide: https://developer.android.com/guide/topics/media/frame-rate

do you think that there is a possibility that this has to be configured by the app? Is it something exo player config will manage? Will it be relevant for Android 12 only?

[tonihei]

ExoPlayer is already using this method, but only on API 30+, so it can't affect the devices showing this problem.

[giladna]

@tonihei if you have any update regarding this we will much appreciate.

[tonihei]

I had a closer look at what happens on my one testing device that reproduces this issue (an Amazon Fire TV Stick) and at least on this device it seems that the decoder is always limiting its output to exactly 50 fps for all videos. That means that videos with higher frame rates and/or increased playback speeds will cause issues because the decoder can't keep up with the required decoding speed. Interestingly, once the frames are "late" (=the presentation time is in the past), the decoder is suddenly able to quickly release all decoded frames, so that no frames are actually dropped, but you get this jerky, jumpy movements in regular intervals.

On this device, setting a codec operating rate (to any arbitrary value) seems to fix the issue and the decoder will decode all samples quickly enough. Could you try the same workaround on your devices to see if this is a general problem on TV boxes? You can force this setting by adding this to your SimpleExoPlayer.Builder (using ExoPlayer 2.15.1):

.setRenderersFactory(
  new DefaultRenderersFactory(/* context= */ this) {
    @Override
    protected void buildVideoRenderers(Context context, int extensionRendererMode,
        MediaCodecSelector mediaCodecSelector, boolean enableDecoderFallback,
        Handler eventHandler, VideoRendererEventListener eventListener,
        long allowedVideoJoiningTimeMs, ArrayList<Renderer> out) {    
      MediaCodecVideoRenderer videoRenderer =
        new MediaCodecVideoRenderer(
            context,
            mediaCodecSelector,
            allowedVideoJoiningTimeMs,
            enableDecoderFallback,
            eventHandler,
            eventListener,
            MAX_DROPPED_VIDEO_FRAME_COUNT_TO_NOTIFY) {
              @Override
              protected float getCodecOperatingRateV23(
                  float targetPlaybackSpeed, Format format, Format[] streamFormats) {
                return 60;
              }
          };
      out.add(videoRenderer);
    }
  })

Please try to make sure to also turn the setLiveMaxPlaybackSpeed(1f) workaround off to actually be able to check if it fixes the problem in all cases.

[giladna]

@tonihei thanks trying that and will get back with my results

10x

[giladna]

@tonihei after applying the code you have suggested on our SDK (and removed the max speed to 1f) we could not see any improvements at all having both solutions you have suggested takes us back to to point where live and catchup works but in case we trigger seek back in the live media dvr window few min back the scrolling text will start having this scroll and stuck scroll and stuck behaviour

why/any chance seek will break the max speed rate behaviour?

Any other workaround you could think of? 10x Gilad.

[tonihei]

after applying the code you have suggested on our SDK (and removed the max speed to 1f) we could not see any improvements at all

Thanks for testing! It's a shame it doesn't work for other devices as well, because it fixes what I think is the same problem on the Amazon Fire TV Stick. I'll try to reproduce on another TV we have in our office once I get to be there again.

why/any chance seek will break the max speed rate behaviour?

This is definitely unrelated to seeking because the setting is only updated when loading the media. I also just tested again to make sure this makes no difference.

[giladna]

thanks Also tried to update the media low latency config directly after seek and it did not have impact.

[giladna]

@tonihei I would much appreciate if you could check the above on your other TV since we got into dead end with this for now.

Is there a chance it might be related to such problem? https://github.com/google/ExoPlayer/issues/2575

thanks Gilad.

[Orenbk]

@tonihei - Appreciate if you can provide an update over this critical issue. According to your previous recommendations (sent to @giladna) we were able overcome the live and catch-up scenarios but left with "Seek" been unresolved. Thank you, Oren.

[markkog]

after applying the code you have suggested on our SDK (and removed the max speed to 1f) we could not see any improvements at all

Thanks for testing! It's a shame it doesn't work for other devices as well, because it fixes what I think is the same problem on the Amazon Fire TV Stick. I'll try to reproduce on another TV we have in our office once I get to be there again.

why/any chance seek will break the max speed rate behaviour?

This is definitely unrelated to seeking because the setting is only updated when loading the media. I also just tested again to make sure this makes no difference.

I would like to understand how you seek accross the media, by looking for the timestamps or based on the IDR frames (sort of trick mode). If according to what you are saying the setting should not be changed when seeking, so we are getting a degrading expirience, someting is not keeping the optimized settings... Woild like to understand the mechanism and why we are having a different experince when rolling back to a normal playback/real time

[tonihei]

I haven't got around to test this on another TV again due to office access restrictions.

Please note that this seems to be a device-specific issue and it's quite likely that there is no good solution to this problem, only potential workarounds that could avoid the issue. My understanding so far is that these devices have some logic in either the MediaCodec implementation or somewhere deeper in the device screen refresh logic (possibly even outside Android) that causes jerky playback (either short frame freezes or sudden unexpected speed ups). Automatic speed adjustment during live playback seems to trigger this problem more often, but it also seems to occur in other situations.

What isn't quite clear to me is how seeking is related to this problem. Could you describe in more detail how you trigger this problem and why it is related to seeking? Does that happen with all streams, or only one particular stream? And does the seek need to happen in a live DVR window to trigger the problem, or does it also occur in on-demand (=not live) content?

[giladna]

@tonihei Thanks for your reply completely understood.

I am not sure it is device specific since I could not see it working perfectly on any box and I also have the feeling it is some how related to the TV or the hdmi cable as well but I do not evidence for that

this issue is mainly observed in this stream with the scrolling text but it can also be observed if there are fast movements in the stream but in usually playback it is not noticeable

in our LowLatency config we limit the min and max speed to be 1.0f in our seek we just call seek api in the dvr window of the live stream we play live+dvr window no VoD in this scenario attaching video here : demo

sending bug reports in mail for the working case and for the broken scrolling text playback case we see after seek

10x Gilad.

[tonihei]

and I also have the feeling it is some how related to the TV or the hdmi cable as well but I do not evidence for that

It would be great if you could test the same device with another TV (ideally a different manufacturer) to verify this.

sending bug reports in mail for the working case and for the broken scrolling text playback case we see after seek

Thanks for providing the reports. Unfortunately, the bug reports don't show anything helpful to debug the issue.

in our seek we just call seek api in the dvr window of the live stream

It's interesting that it only seems to occur after a seek. This points to either a performance issue where the seek causes something to constantly lag behind or to a codec issue where the seek (or rather the codec flush) puts the codec in a problematic state.

To further understand and analyze this problem, you try out a few things:

• Try pausing the stream after the seek and then resume to see if this has any influence on the issue.
• Try forcing a full codec reset when seeking. You could fork MediaCodecVideoRenderer and return true from codecNeedsFlushWorkaround or just call player.stop() and then re-prepare playback at the seek position. When forking MediaCodecVideoRenderer, you can provide your fork in a subclass of RenderersFactory to the player.
• If you control the content creation, it would be helpful to see what happens when the content is re-encoded at other frame rates (like 60 fps or 30fps), to see if this is specific to 50fps content.
• Try using the media codec's async mode, which may give some performance improvement. In ExoPlayer 2.15.1, you can do this by setting experimentalSetAsynchronousBufferQueueingEnabled(true) on the DefaultRenderersFactory.
• Try disabling the frame output timing by passing 0 to releaseOutputBuffer (see https://github.com/google/ExoPlayer/issues/6348#issuecomment-539061260). This also requires forking MediaCodecVideoRenderer.

None of these things may actually help or these changes are not necessarily a good idea for production code, but if you see a difference, this may help to find a better long-term solution.

[markkog]

Hi

and I also have the feeling it is some how related to the TV or the hdmi cable as well but I do not evidence for that

It would be great if you could test the same device with another TV (ideally a different manufacturer) to verify this.

sending bug reports in mail for the working case and for the broken scrolling text playback case we see after seek

Thanks for providing the reports. Unfortunately, the bug reports don't show anything helpful to debug the issue.

in our seek we just call seek api in the dvr window of the live stream

It's interesting that it only seems to occur after a seek. This points to either a performance issue where the seek causes something to constantly lag behind or to a codec issue where the seek (or rather the codec flush) puts the codec in a problematic state.

To further understand and analyze this problem, you try out a few things:

• Try pausing the stream after the seek and then resume to see if this has any influence on the issue.
• Try forcing a full codec reset when seeking. You could fork MediaCodecVideoRenderer and return true from codecNeedsFlushWorkaround or just call player.stop() and then re-prepare playback at the seek position. When forking MediaCodecVideoRenderer, you can provide your fork in a subclass of RenderersFactory to the player.
• If you control the content creation, it would be helpful to see what happens when the content is re-encoded at other frame rates (like 60 fps or 30fps), to see if this is specific to 50fps content.
• Try using the media codec's async mode, which may give some performance improvement. In ExoPlayer 2.15.1, you can do this by setting experimentalSetAsynchronousBufferQueueingEnabled(true) on the DefaultRenderersFactory.
• Try disabling the frame output timing by passing 0 to releaseOutputBuffer (see 60 FPS Widevine DASH content stutters on some devices #6348 (comment)). This also requires forking MediaCodecVideoRenderer.

None of these things may actually help or these changes are not necessarily a good idea for production code, but if you see a difference, this may help to find a better long-term solution.

As I can see from the the video https://drive.google.com/file/d/1Esuwv8MQJQ6zZdrKisEQ6Q6hhMDHtdbB/view?usp=sharing the seeking function drag the curssor to a specific time code on the line where the video starts from that point. It is not like a key frame jugger (trick mode) where the timestamps are know in advance and accurate where you actually skip between the keyframes. in this scenario you can not start from every point and play from every frame, the mechanism looks for a closer keyframe or frame reference. on that point the timestamp sequence might not be accurate when playing from that specific point and especially when rolling back to live playback mode. Few points that you've mentioned are important to check, rendering, codec refresh and output buffer. Please clear about the seeking mechanism behind the operation. thank you Mark

[tonihei]

This issue is also likely to be a duplicate of #7683 and it's also worth trying the suggestion in https://github.com/google/ExoPlayer/issues/7683#issuecomment-699036483 (in addition to the suggestions above).

[giladna]

@tonihei

I have forked the code as you have request and combined both
experimentalSetAsynchronousBufferQueueingEnabled(true) + Try disabling the frame output timing by passing 0 to releaseOutputBuffer

I could see that there is some improvement when seeking (it will have the issue for several seconds and then it becomes fine)

that is better than it was but still not perfect

10x Gilad

[tonihei]

Thanks for checking!

I have forked the code as you have request and combined both experimentalSetAsynchronousBufferQueueingEnabled(true)

The fact that experimentalSetAsynchronousBufferQueueingEnabled(true) helps very likely indicates a performance problem with the high frame rate streams on these devices (as already suggested above). Using the async mode makes the rendering process a but more performant, but there's generally not much we can do if the device just isn't capably of decoding the video fast enough.

Try disabling the frame output timing by passing 0 to releaseOutputBuffer

The fact that this seems to help again points to some device-specific logic outside of our control that tempers with the frame release timestamps (or automatically drops frames). We can keep this issue open to test out potential work workarounds / adaptations to these devices. It's unlikely we get around to this soon though.

that is better than it was but still not perfect

Have you considered changing the track selection to only selects tracks with a lower frame rate where the device is capable of decoding them fast enough? You can, for example, specify TrackSelectionParameters.Builder.setMaxVideoFrameRate(30) if you know a certain device isn't capable of playing 50 or 60fps smoothly.

[Kuphi]

I had a closer look at what happens on my one testing device that reproduces this issue (an Amazon Fire TV Stick) and at least on this device it seems that the decoder is always limiting its output to exactly 50 fps for all videos. That means that videos with higher frame rates and/or increased playback speeds will cause issues because the decoder can't keep up with the required decoding speed. Interestingly, once the frames are "late" (=the presentation time is in the past), the decoder is suddenly able to quickly release all decoded frames, so that no frames are actually dropped, but you get this jerky, jumpy movements in regular intervals.

This issue currently also exists in jellyfin-androidtv. I tried to add the code to the VideoManager.java file, but wasn't able to build the apk. Can you help to transfer this solution to jellyfin-androidtv? Thank you very much!

https://github.com/jellyfin/jellyfin-androidtv/issues/3881