Significant video media flashing/artifacting under certain conditions with 'Use Low Latency' (see desc)

[techanon]

Describe the bug (This issue was observed within the VRChat platform, but I don't think it's a VRC problem based on my testing and analysis of the issue) I am observing a particular visual issue when trying to watch a twitch or dlive livestream (the issue is NOT present when watching a youtube livestream though) at a med-high resolution using the Use Low Latency option. Audio is NOT affected (videos below demonstrate). And example of the issue is when the stream starts playing on 1080p resolution cap, it immediately starts doing 1 frame/sec flashing (see videos for visual) with a black screen most of the time. Then, after some indeterminate amount of time, it will start artifacting (see 2nd video).
If the stream plays on 720p res cap, it will have minor color artifacting horizontally across the video at a specific vertical position every time (see 1st video).
If the stream plays on 480p res cap, it will flash for a bit (like during the 15 second "commercial break" lead-in part of the twitch stream), but then eventually produce perfectly fine video (see 1st video).

My current analysis leans towards it being an AMD GPU (maybe my specific one?) issue.
I had also requested the help of a couple friends, one who had a RTX 3090 and one who had a 1080Ti, to see if they saw the problem. Both of their tests followed the same steps and had no notable visual problems. I have not been able to find someone else at this time with an AMD GPU different than mine to test if it's an AMD problem in general. I have updated my drivers to the latest (as screen shot below shows) and the issue is still present. Also, when I disabled the Use Low Latency option, the video plays 100% normal on my system. I need the low latency option for handling close to source streams for group events that take place on the VRChat platform.

Your Setup (please complete the following information):

• Unity version:
  • 2018.4.20f1 (required by VRChat)
• AVPro Video version:
  • Log output from VRChat:
  • [AVProVideo] Initialising AVPro Video v2.0.7 (native plugin v2.0.7f1-ultra) on Radeon RX 580 Series/Direct3D 11.0 [level 11.1] (MT False) on WindowsPlayer
• Operating system version:
  • Win 10 64 bit (version 2004)
• Device model:
  • CPU: AMD Ryzen 5 3600X
  • GPU: Radeon RX 580 (see screenshot below for specific versions)
• Video specs (resolution, frame-rate, codec, file size):
  • Varying resolution caps tested (480/720/1080)
  • Twitch/DLive streams (not sure of frame-rate/codec/etc)
  • Youtube streams (not sure of frame-rate/codec/etc)

To Reproduce

1. Have AMD GPU? (in my case a Radeon RX 580)
2. Set the resolution cap to 1080 or above on the AVPro Video Player component and enable the Use Low Latency option
3. Attempt to play a Twitch stream (also applies to DLive streams, but not Youtube streams)

Logs No related logs other than the version numbers above

Screenshots

• Specific versions listed in the Radeon control panel software

Videos This video shows the difference between a YT stream and a Twitch streams as well as the difference between resolution caps for a Twitch stream (due to difference in game fps and recording fps, some flashing frames were missed in the recording):
https://streamable.com/qm1zse
This video shows the two kinds of glitches that occur - The flashing, followed by (after a random amount of time of the video playing) artifacting glitches: https://streamable.com/fiw0gp

The dropdown UI element seen in the 1st video simply swaps the enabled player to one with the corresponding resolution.
There is no difference between the video players other than the max resolution setting.

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Instead email the link to us unitysupport@renderheads.com

---

This was a bit of a convoluted bug to collect info on, so I may have missed something. Please let me know if I did.

[kahnivore]

Hi @techanon

Thanks for your detailed post.

We'd have to find a way to reproduce this, but don't have access to an AMD card currently.

The black frames are strange, but some amount of artifacting is to be expected with low-latency.

There's also a chance that this might be fixed in our next release 2.1.0, which will be out very soon.

It would probably be best to try updating to 2.1.0 when it's out as a first point of call.

Thanks

[kahnivore]

2.0.10 has been released. Could you let us know if this update makes a difference?

Thanks.

[techanon]

I will have to wait until my upstream provider (VRChat) updates on their end, but will let you know once I've been able to test it with the new version.

[techanon]

I was given advice on how to test the tool independent of VRChat via trial version. After doing so, I can confirm that with 2.0.10, the issue is still present. Though it doesn't affect ALL streams. I found one random one that appears to only deal with the green artifacts (which is known and fine).

Repro steps:

• Import 2.0.10's Media Player example from the trial version
• Set the Use Low Latency option to true under the Windows settings section
• Find a twitch stream URL that is live
• Use Youtube DL to resolve to a m3u8 direct media url (I cheat and use VRChat directly for this)
• Insert direct media url into the path URL input for the media player
• Click unity Play button

I did this process for multiple streams that were occurring during my testing. I only found one that was affected minimally or not at all while Low Latency was active, though I haven't been able to come up with an idea of why...

Here is a video demonstrating the differences: https://streamable.com/wc3hqz

If you think the contents of the m3u8 will help, I can get those as well.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. If you wish to continue this issue, then please create a new issue as we do not monitor closed issues.

[JPT580]

Unfortunately, this issue was closed due to inactivity. However, I believe that the issue still persists.

Let me provide a bunch of data collected to help tracking it down further. (Please excuse the mostly german labels in my screenshots - feel free to ask for clarifications if required.)

My setup

I am running a fairly modern AMD system with the current stable version of radeon driver software (as well as Windows 10). Here are details provided by the radeon driver software:

Additional notes

• As already described above, audio playback is never affected.
• During the experiments, my GPU was at about 60-70% usage and my CPU was somewhere between 30-60% usage (with enough headroom across all cores, so no core was spinning at 100%)

Case study: Playback of a Twitch stream in a busy (10+ players) VRChat world

Here are details provided from Twitch in Firefox regarding the Twitch stream used:

Playback was done in a VRChat World called "Club Orion". The following snippet shows debug logs that show youtube-dl in action, resolving the Twitch URL to a usable ressource url:

Once playback is started, a bunch of flickering occurs between a black screen and the occasional frame from the Twitch stream. At some point, video playback begins, but video stuttering (alternating between fresh playback frames and "old" frames from past playback) occurs.

During playback, partially corrupted frames (notice the green in the bottom right corner) appear.

After some time, a completely corrupted frame joins the stuttering effect, which I managed to capture as follows:

A short video demonstrating the glitch as observed in vrchat (desktop mode) can be found here: https://streamable.com/2ptzzt (Audio was trimmed due to people talking, but I assure you audio playback is fine.)

My personal impression about this issue

I did another experiment with the following differences:

• I used a random Twitch stream
• I used an empty instance of "Club Orion" to try and reproduce the issue on my own.

Under these circumstances, I "only" got occasional playback stuttering (looked like small fragments of the video were on repeat). I also observed the "horizontal green line" as described above. It seems like it takes more time to pass for the glitch to manifest when I was alone in the world.

To put it the other way around: The glitch manifests way faster if more than one (in my case 10+) players are within the world.

Brainstorming/Ideas

The green line and also the fully corrupted frame I managed to capture strongly remind me of artifacts that occur when video decompression goes wrong. (I saw similar artifacts when trying to play corrupted video files.) Therefore, I suspect that one possible cause could be some sort of memory management/corruption issue (like a memory leak corrupting nearby data).

Another possible explanation could be that there is an issue in how the video playback logic interacts with the rest of the game engine. I already tried launching vrchat with --fps=240 to see if that somehow fixes the issue, but it did not.

This is where I'm out of ideas and things to try. If there is something I could try to provide further data, please let me know.

[AndrewRH]

In the latest version (2.1.8) there are some options you can disable such as 'stereo support' and 'hap/notchlc support' - these may help your issue.

I haven't been able to to reproduce this issue, but it looks like it's related to the GPU decoder being worked too hard - perhaps check in Task Manager > Performance > GPU tab to make sure the GPU video decoder isn't too high.

Thanks,

[techanon]

I have made additional comments in the above mentioned thread ( #940 )

[AndrewRH]

Thanks for the additional comments.

We've had reports of this happening using Twitch and YouTube streams (both of which we do not support nor encourage usage of since it is against the TOC's), but the issue seems to only happen in Low Latency mode. Which leads us to believe that is it just a bug/quirk in the Microsoft decoder when in this mode which favours low latency over quality....

[AndrewRH]

Unfortuntely there is little we can do about this. We're using the low latency implementation from Microsoft's Media Foundation as documented here: https://docs.microsoft.com/en-us/windows/win32/medfound/mf-low-latency It seems that this option favours speed over quality, and perhaps has a bug. There's nothing we can do about this on Windows as it's built-in.

This issue is now closed and we recommend trying to use the WinRT video API option as it seems to work better.