Native Video Player

[perrinpagess]

Describe the feature you'd like

I want an option to use AVPlayer for direct HLS instead of going through VLCKit. A toggle under "advanced settings" would be nice because it has some limitations.

Additional context

AVPlayer uses Apple's "EDR" tone mapping implementation for HDR playback on SDR and HDR displays. AVPlayer's HDR tone mapping is substantially better quality than VLCKit. AVPlayer natively supports surround audio for spatial audio processing and head tracking. Picture in picture support is native with AVPlayer as well. AVPlayer's surround to stereo mixdown is better than anything ffmpeg can do (from my testing).

I made a rudimentary video player in swift playgrounds to help demonstrate what I want (see screenshot). Instead of the url I used for the demo it would be the HLS url for a video on my Jellyfin server.

Thanks!

[LePips]

I did implement the native player during my video player refactoring however I removed it due to navigation issue with selecting subtitles and such. It was just too much work at the moment.

We also need to build the proper direct play profiles for using the native player. That will push this back a lot.

I have thought to re-implement it and the work is in git history. This however made me think to create another setting where the AVPlayer layer is the rendering view for the video and we can use the same VLCKit overlay.

I'll first re-implement the native players as a part of this.

[perrinpagess]

What was the issue with subtitles? My demonstration had the ability to pick subtitles with no extra work. Safari's current implementation of the native video player allows me to pick subtitles as well. HLS specification requires WebVTT subtitle format.

What are direct play profiles required for? All my video files are sanitized and use Apple's HLS authoring approved codecs in MP4 container with proper headers. My MP4 files stream flawlessly in Safari when I copy a link from Jellyfin.

[LePips]

There's a lot at play here. Most people's media aren't nice and the webvtt subtitle streams aren't consistent from the server, so we have to build the native player around that. For most, this would force embedding subtitles that they would choose before they play the media. Or in your case, you would be able to choose not to embed any subtitles and choose to play them from the stream instead.

[perrinpagess]

I understand that most people don't want to learn how to use ffmpeg to make their files HLS compliant. I suggested hiding a toggle in "advanced settings" or whatever so that you wouldn't have to do all the extra work to make native player play nice with non-HLS compliant files.

I am fine using Safari for video streaming; however, my original intent was to suggest an enhancement that's only useful for advanced users to keep initial development work to a minimum. The native video player function could be improved in the future or just kept barebones.

[LePips]

Of course I'll provide the option.

This has me thinking though, people might prefer the native video player, especially on tvOS since it feels a bit nicer.

Are there tutorials on making content hls compliant? That would make a great plugin

[perrinpagess]

https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices

I don't know of any tutorial...I had to figure out all this stuff myself. I am willing to put in time to help make a plugin. The link above is a good write up by Apple. It's a necessary read (IMO), but also lengthy so I can give some key points.

• HEVC video must be tagged as hvc1. This is possible with the ffmpeg command "-tag:v hvc1"
• Dolby Vision must be profile 5. Profile 7 (the kind on UHD Blu-ray) is not supported. I haven't found a way to convert between profiles yet. I don't think there is a way without buying software from Dolby. Just remove the Dolby Vision video stream from your files for HLS, it's wasting space otherwise.
• AC3 for 5.1 audio and EAC3 for 5.1 or 7.1 audio. AAC for stereo. ffmpeg's native EAC3 and AAC encoders are terrible so only encode audio with its AC3 encoder or Fraunhofer FDK AAC encoder. TureHD is not yet supported on Apple devices, but Atmos is with EAC3 format (ffmpeg doesn't support manipulating Atmos tracks though so they must be untouched). Original AC3 or EAC3 audio is preferred over encoding from TrueHD or DTS HD-MA because commercial Dolby encoders are much better than ffmpeg. ALAC and FLAC multichannel are supported, but those have limitations in the MP4 container format and high bitrate lossy audio is transparent anyway.
• The video and audio bitrates given by Apple on their HLS specification website are more of a suggestion for companies who pay for streaming bandwidth. HLS doesn't need the video to be transcoded to a lower bitrate. The website lists video/audio codec profile and level compatibility per SOC in a table.
• PGS subtitle formats are not supported. Subtitles must be text-based in WebVTT format. Burn in subtitles only as a last resort, get text-based subtitles from opensubtitles if possible.

[LePips]

Thanks! I do think a plugin would be awesome in the long run.

Would you be able to supply me a script for now for my own testing?

[perrinpagess]

Sure, every video file has a different video/audio/subtitle track order so you will have to use ffprobe and the "map" command to make sure you're copying/encoding the correct tracks. Remove "-tag:v hvc1" from the script if you're working with H.264 video or else ffmpeg will give an error. Video transcoding takes ages, the quality is never as good as the source, and my networking setup is capable of gigabit speeds through wifi, so I don't bother with it.

I like using this video for testing HDR10: https://4kmedia.org/sony-swordsmith-hdr-uhd-4k-demo/

This Dolby Vision file works with no manipulation: http://media.developer.dolby.com/DolbyVision_Atmos/mp4/iOS_P5_GlassBlowing2_3840x2160%4059.94fps_15200kbps.mp4

If your audio and subtitles are already in a compatible format and it's an HEVC video stream: ffmpeg -i input.mkv -c copy -tag:v hvc1 -map 0:v:0 -map 0:a:0 -map 0:s:0 -map_chapters -1 output.mp4

If audio isn't in a compatible format (something like dts and there's no secondary AC3 stream) and it's an HEVC video stream: ffmpeg -i input.mkv -c:v copy -tag:v hvc1 -c:a ac3 -b:a 640k -c:s copy -map 0:v:0 -map 0:a:0 -map 0:s:0 -map_chapters -1 output.mp4

If audio and subtitles aren't in a compatible format and it's an HEVC video stream: ffmpeg -i input.mkv -c:v copy -tag:v hvc1 -c:a ac3 -b:a 640k -c:s webvtt -map 0:v:0 -map 0:a:0 -map 0:s:0 -map_chapters -1 output.mp4

[holow29]

This has me thinking though, people might prefer the native video player, especially on tvOS since it feels a bit nicer.

Just wanted to say as a nobody following development here, definitely would prefer native video player on tvOS (which is my main use case for Swiftfin). It was one reason I was excited to hear about Swiftfin in the first place :) I haven't delved into any technical limitations with it, but it seems like the more native an app is on tvOS, the better experience it provides.

[LePips]

I'm going to leave some developer notes for others who will want to tackle this sometime

https://github.com/jellyfin/Swiftfin/discussions/324