[Bug] Picture disintegrates when left idle

[akechi-haruka]

Describe the bug When some games are left idle in certain spots and only have minimal changes between the frames, the stream first starts to become choppy, and after a few seconds completely disintegrates into a glitchy mess. As soon a "bigger" change happens, like moving the camera the smallest bit, this immediately fixes itself. This is not related to input, if I press anything that doesn't change the frame, it doesn't fix itself. This only happens on certain games, and sometimes only in certain spots interestingly.

My setup is otherwise fine via LAN Adapter and I'm not having any delay, noticeable latency or other instability issues and hit timed sections or generally play without issues (Major thank you for this!).

Occurs on (tested):

• Paper Mario: The Origami King (Toad Town Main Square anywhere, NOT on Picnic Road but in MENUS in Picnic Road)
• Disgaea 1 Refine/Complete (Base Room 2, Tutorial Map 1)

Video (watch the Save Block animation, it keeps recovering a few times but it gives up at 0:40+): https://mega.nz/embed/ogoyFZQD#qzWjVDZAeQ3946HTlh69658wDWHu1iWPbs3qnPdiGI4

Does NOT occur on (tested):

• Super Mario Odyssey (anywhere)
• Taiko no Tatsujin: Nintendo Switch Version (menus; gameplay has recording disabled)
• Fairune Collection (anywhere)

To Reproduce Launch any of the affected games, going to the areas I've tested, and cease any input for ~10 seconds for the first effects to appear.

Expected behavior No corruption when left idle.

Setup information

• PC Os: Windows 10 x64
• Console firmware: 10.2.0
• Custom firmware name and version: Atmosphere 0.15.0
• SysDVR version: 4.0.1

Additional context No other sysmodules

Launch command line:

start dotnet SysDVR-Client.dll bridge 10.0.0.103 --mpv C:\ProgramData\chocolatey\lib\mpv.install\tools\mpv.com --profile=low-latency --no-cache --cache-secs=0 --demuxer-readahead-secs=0 --cache-pause=no --untimed --quiet --really-quiet --no-input-default-bindings
start dotnet SysDVR-Client.dll bridge 10.0.0.103 --mpv C:\ProgramData\chocolatey\lib\mpv.install\tools\mpv.com --no-video --profile=low-latency --no-cache --cache-secs=0 --demuxer-readahead-secs=0 --cache-pause=no --untimed --quiet --really-quiet --no-input-default-bindings

This also happens with and without --untimed, as well as with synchronized mode, launched with

start dotnet SysDVR-Client.dll bridge 10.0.0.103 
sleep 3
mpv rtsp://127.0.0.1:6666/ --profile=low-latency --no-cache --cache-secs=0 --demuxer-readahead-secs=0 --cache-pause=no --untimed --quiet --really-quiet --no-input-default-bindings

and turning on mpv log does not give anything interesting:

[exelix11]

I can confirm i've seen this behavior, even in mario odyssey as i mainly test on that, As i'm typing this i'm not able to reproduce it but yes it's there and i don't know why. I assume it has to do with the video player but couldn't nail down anything in particular that tiriggers it. For the next update (which is coming out very soon) i've implemented a new video player and didn't notice it while testing so hopefully that solves it. Still will keep this open as may likely come back again and it's relevant for mpv users.

[lKomus]

I've seen this in 4.02 as well. There's also a game called La-Mulana that is unstreamable since the game is 90% 2D backgrounds that don't move, so you get these kind of glitches all the time, even when you just play normally.

[nzxth2]

I can reproduce this as well, on SysDVR version 5.0.2 (FW 10.1.0, AMS 0.14.2), using Monster Hunter Generations Ultimate.
It's still exactly as described above; when leaving the game idle on certain scenes that are mostly static, the video immediately becomes choppy and the image starts glitching out.
In addition, the image gets worse the longer you leave it on idle (see images) and only fixes itself momentarily when the image changes drastically, e.g. by moving the character/camera or opening a big menu box.
Otherwise I'm experiencing no issues at all, streaming perfectly fine through USB.

Edit: Perhaps this section on the troubleshooting page is relevant? That part about the console not emitting a keyframe when the screen is mostly static, until the game camera is moved.

Apologies for not being able to provide further help. I googled a bit and read here that "x264 will, by default, create a keyframe when it detects a scene change". So perhaps it fails to detect a scene change, maybe because the threshold is too high? The article goes on to say that you can use the no-scenecut option to turn off scene detection in libx264 (or by using the FFmpeg -sc_threshold 0 parameter). Not sure if this is of any help, but I wanted to mention it.

[hexindent]

Aviary Attorney has this problem too (SysDVR 5.0.2, FW 11.0.0)

[Sch1nken]

Just chiming in to reiterate what @nzxth2 said. It's somewhat related to scenecut, though that's only a sideeffect and no real solution.

I've recently dabbled in realtime video with both h264 and vp8 and hat these very same artifacts (though on vp8 codec). For live video, especially for h264 you want the SPS and PPS to be sent with every IDR frame. This is the major fix that made it work for me (note: completely different, not even switch related application).

Also I had a bug where I read packets twice but only sent out one of them (effectively dropping A LOT of packets), this was really only noticeable when there was minimal motion (like in the cases mentioned above).

Edit: Skimming through the code you're using RTSP, i was using plain RTP but this is somewhat related it seems. I'll continue going through the code and see if I can find something.

[exelix11]

Unfortunately i have no control on the actual encoding process that's done with the console hardware.

For live video, especially for h264 you want the SPS and PPS to be sent with every IDR frame

I already do this every once in a while before sending NALs but it's at random, i could try identifying IDR frames though.

Edit: Skimming through the code you're using RTSP, i was using plain RTP but this is somewhat related it seems. I'll continue going through the code and see if I can find something.

RTSP is just a control protocol for RTP, actual video data is sent through an RTP session that's established via RTSP, but that is not the problem here as the glitch happens with different transports as well, including manually feeding the video data to ffmpeg decoder with the built-in player

I'm currently working on an mp4 exporter which can hopefully help with this issue as it will allow me to test this behavior in different players and to use h264 analysis tools.

[exelix11]

Did more research on this, the console seems to emit an IDR frame every ~2 seconds except when the screen in still (and only in certain conditions it seems). When the screen is still the console will only emit non-IDR frames no matter what and that's what causes this issue, now i would assume this is a configuration issue in ffmpeg cause i don't really see why straming only non-IDR frames would break the stream assuming at least one IDR frame has been received successfully. This seems to be common practice for low latency applications but unfortunately I couldn't find any explaination for this behavior yet.

[exelix11]

Turns out, the console is emitting IDR frames just fine, it's just that the default buffer size is too small so they won't get copied, increasing that fixed the issue. This should not happen anymore in latest release