[Feature] A17 Pro AV1 Decode Support

cryptofyre commented 7 months ago

Is your feature request related to a problem? Please describe. While this request is more pertinent for users with less robust internet connections, it is not strictly tied to an issue within the Moonlight app itself.

Describe the solution you'd like Recently Apple has announced AV1 decoding support for the iPhone 15 Pro series as part of the A17 Pro chip, having currently owned both the NVIDIA RTX 4090 which is capable of encoding and decoding AV1 and the iPhone 15 Pro Max which is now capable of decoding AV1 on a hardware level this would make the most sense to take advantage of considering it's now available in Sunshine's nightly branch for support and may be released to stable soon.

Moonlight could accommodate the new codec on these devices during stream time similarly to how the HEVC decoder currently works in the pre-existing app. This would allow people with lesser known and powerful connections to have a good experience given the effectivity of the codec and would allow for a very smooth and optimal stream for the user especially in Cellular and Rural environments.

Describe alternatives you've considered The existing HEVC codec does a decent job at streaming but shows limitations in handling "active" frames, leading to quicker pixelation. AV1 would offer a more stable and superior streaming experience across various types of content and network conditions.

Additional context A17 Pro AV1 Reddit Post Apple's official announcement Sunshine AV1 Discussion

While I can't find any existing documentation for this new silicon feature yet as part of developer.apple.com 's AVFoundation documentation but I'm sure someone will have some lesser-known documents on it soon, if not it very well could be already accessible using pre-existing methods.

cgutman commented 7 months ago

I am ready to add it whenever documentation or example implementations are available. I have already stubbed out some of the code, but I will probably need help from a developer with an iPhone 15 Pro to complete and test the implementation.

Petkov0721 commented 7 months ago

Hey, I have an iPhone 15 Pro Max. If you need me to test anything, I would be glad to help! :)

cgutman commented 7 months ago

Do you have a Mac and Xcode?

Petkov0721 commented 7 months ago

I do yes. I have some experience with Swift and Objective-C as well so if you need any help just let me know

cgutman commented 7 months ago

OK, assuming you are running Sunshine v0.21 with a GPU that has AV1 encoding support, try this:

First confirm that AV1 encoding is working from your host with another Moonlight client (the desktop client supports software decoding AV1 if you don't have another client with an AV1 decoder)
In Xcode, you will probably need to change the application ID to something different to avoid conflicting with the real App Store version (which will cause signing errors)
Replace all these lines https://github.com/moonlight-stream/moonlight-ios/blob/f20d90791a3657bd12941cb6572fa6fdaa42e12a/Limelight/ViewControllers/MainFrameViewController.m#L666-L676 with _streamConfig.supportedVideoFormats = VIDEO_FORMAT_AV1_MAIN8;
Enable the performance overlay to confirm AV1 is being used properly
Try streaming with the modified build

If it works, you should see a picture. If not, you will probably just get a black screen with performance stats displayed (which should say AV1).

Petkov0721 commented 7 months ago

So I did all that. I am on the master branch with the latest commits pulled and changed the lines to set it to AV1. I have a 4090 host with the latest Sunshine 0.21. I am getting a black screen. The performance stats displays AV1 (will attach a screenshot) but the screen is black. Also to confirm that it works, with the lines set to the default code it works fine.

Tomorrow I will try to play with it to see what it might be causing it and if I can help with a solution.

Petkov0721 commented 7 months ago

So a few notes after another testing attempt. Neither VIDEO_FORMAT_AV1_MAIN8 nor VIDEO_FORMAT_AV1_MAIN10 work - which is to be expected. Black screen again. Touch control and gamepad buttons work so input is submitted.

When running with either MAIN8 and MAIN10 and the screen is on the black screen with the performance overlay, xCode is reporting this error constantly:

Waiting for IDR frame
Waiting for IDR frame
Waiting for IDR frame
Waiting for IDR frame
Waiting for IDR frame
Waiting for IDR frame
Waiting for IDR frame
Network dropped 1 frame (frame 65)
Waiting for IDR frame
Unrecoverable frame 67: 24+0=24 received < 74 needed
IDR frame request sent
Unrecoverable frame 68: 7+0=7 received < 74 needed
Unrecoverable frame 74: 3+0=3 received < 50 needed
Network dropped 8 frames (frames 67 to 74)
Waiting for IDR frame
IDR frame request sent
Waiting for IDR frame

<WARN> Connection status update: 0

nw_http2_connection_receive_goaway_block_invoke [C14.1.2.1.1:1] stream in node 0x282643660 in id table will remain active because its stream id (1) is less than 1, skipping

Unrecoverable frame 594: 49+0=49 received < 52 needed
Network dropped 1 frame (frame 594)
Waiting for IDR frame
IDR frame request sent
Unrecoverable frame 596: 17+0=17 received < 52 needed
Unrecoverable frame 597: 43+0=43 received < 52 needed
Network dropped 2 frames (frames 596 to 597)
Waiting for IDR frame
IDR frame request sent
Waiting for IDR frame
Unrecoverable frame 1038: 51+0=51 received < 58 needed
Unrecoverable frame 1039: 35+0=35 received < 58 needed
Unrecoverable frame 1040: 31+0=31 received < 58 needed
Network dropped 3 frames (frames 1038 to 1040)
Waiting for IDR frame
IDR frame request sent
Waiting for IDR frame
Waiting for IDR frame
Waiting for IDR frame
Unrecoverable frame 1045: 67+0=67 received < 84 needed
Network dropped 1 frame (frame 1045)
Waiting for IDR frame
IDR frame request sent
Waiting for IDR frame

Just to confirm, running the original code or even with just this line: _streamConfig.supportedVideoFormats = VIDEO_FORMAT_H264; everything works as expected.

I have also tried switching HDR on and off on my PC display with both options experiencing the same issue.

Everything is set to the default option in my Sunshine config.

During the weekend I will have more time to debug hopefully, but it would be helpful if you could point me in which direction to look. Thank you!

cgutman commented 6 months ago

Thanks for investigating. I pushed another change in 0a59ce0ca908d3f6231d7434e50172bbceba6263 that might fix AV1. If you could give that a try, that would be great.

I suspect though we'll need to provide a av1C codec metadata blob per https://aomediacodec.github.io/av1-isobmff/#av1codecconfigurationbox-syntax (which is similar to what is required for VP9 with VideoToolbox). Chrome does the VP9 stuff here: https://source.chromium.org/chromium/chromium/src/+/main:media/gpu/mac/vt_config_util.mm;drc=977dc02c431b4979e34c7792bc3d646f649dacb4;l=155

Petkov0721 commented 6 months ago

I tested it again with the new changes and got pretty much the same result. Here is the log:

Waiting for IDR frame
Waiting for IDR frame
Waiting for IDR frame
Recovered 1 video data shards from frame 55
Waiting for IDR frame
Waiting for IDR frame
Recovered 1 video data shards from frame 57
Waiting for IDR frame
Unrecoverable frame 58: 3+0=3 received < 78 needed
Unrecoverable frame 59: 24+0=24 received < 77 needed
Unrecoverable frame 60: 2+0=2 received < 64 needed
Unrecoverable frame 61: 41+1=42 received < 80 needed
Unrecoverable frame 62: 20+0=20 received < 69 needed
Unrecoverable frame 63: 3+0=3 received < 70 needed
Unrecoverable frame 65: 28+1=29 received < 66 needed
Unrecoverable frame 66: 15+1=16 received < 47 needed
Unrecoverable frame 70: 1+0=1 received < 44 needed
Network dropped 13 frames (frames 58 to 70)
Waiting for IDR frame
IDR frame request sent
Waiting for IDR frame
Unrecoverable frame 73: 36+0=36 received < 55 needed
Unrecoverable frame 74: 13+1=14 received < 55 needed
Unrecoverable frame 75: 12+0=12 received < 53 needed
Unrecoverable frame 76: 39+0=39 received < 54 needed
Unrecoverable frame 78: 8+0=8 received < 59 needed
Unrecoverable frame 79: 1+0=1 received < 51 needed
Unrecoverable frame 80: 16+0=16 received < 55 needed
Network dropped 8 frames (frames 73 to 80)
Waiting for IDR frame
IDR frame request sent
Unrecoverable frame 82: 17+0=17 received < 32 needed
Unrecoverable frame 83: 1+0=1 received < 41 needed
Unrecoverable frame 84: 5+1=6 received < 40 needed
Unrecoverable frame 85: 16+0=16 received < 50 needed
Network dropped 5 frames (frames 82 to 86)
Waiting for IDR frame
IDR frame request sent
Waiting for IDR frame
Recovered 2 video data shards from frame 90
Recovered 1 video data shards from frame 97
Recovered 1 video data shards from frame 102

I started looking at the VP9 implementation and how av1C should be implemented, but I haven't had much time to test anything. If everything goes well this weekend I might have more free time to look at it, but my knowledge of video/audio codecs is pretty limited, so it will probably take me some time. Still a very nice learning experience though.

cgutman commented 6 months ago

I added support for setting the AV1CodecConfigurationBox using FFmpeg in https://github.com/moonlight-stream/moonlight-ios/commit/b81ea6b8743651412190ae102c6754d52ef94ae7. This seems to be what Chrome's new AV1 support is doing too in https://chromium-review.googlesource.com/c/chromium/src/+/5004249

@Petkov0721 Please try the latest code and see if AV1 works now.

Petkov0721 commented 6 months ago

I confirm that it works flawlessly now. Both HDR and non-HDR formats and the latency is better compared to HEVC. I am still testing it, but so far it seems to work perfectly!

cgutman commented 6 months ago

@Petkov0721 Great to hear! If you don't mind, please try switching between HDR enabled and disabled a few times in your PC graphics settings while Moonlight is streaming (with the HDR option enabled) just to make sure Moonlight properly handles that SDR-HDR transition.

Petkov0721 commented 6 months ago

Hey, so a few things after testing quite a lot. First (I attach screenshots), when I turn off the HDR in Windows while the stream is on, the Codec in the Performance Overlay changes to AV1 10-bit SDR as expected, but the screen gets very bright and overexposed. This does not happen when using HEVC. Second, sometimes (more often than not) when connecting with AV1 it takes 2-5 seconds for the stream actually to start. I get the initial frame and it shows what is on the screen, but input and the stream itself is frozen for 2-5 seconds until it settles and everything works well after that. Third, once during a few hours of testing, the stream started having latency issues and dropping frames - which sometimes happen as I am on Wi-Fi, but this time I got a Connection Error dialog box, but the stream did not stop and after a few seconds everything normalised and the stream continued even though I clicked Ok on the box. This might be happening with HEVC as well, not sure, but it has never happened like that before.

cgutman commented 6 months ago

OK, I think I fixed the HDR switching in https://github.com/moonlight-stream/moonlight-ios/commit/089b424f658b8e02049dc632f72919c160845500. We forgot to create a new CMVideoFormatDescription when switching in and out of HDR.

Can you test that scenario again?

Petkov0721 commented 6 months ago

Hey, yes - this is now fixed. The transition between HDR and non-HDR is seamless and looks exactly like the HEVC variant (it is no longer overexposed). Surprisingly, the initial 2-5 lag that I reported with AV1 is gone as well. I don't know what caused it in the first place but now it seems like it is working perfectly. Also, I still haven't experienced the third scenario a second time, so it might have been related to my connection. I will continue testing AV1, but I think it works great so far!

cgutman commented 5 months ago

This feature has shipped in v9.0.0.

moonlight-stream / moonlight-ios

[Feature] A17 Pro AV1 Decode Support #585