Native Player Fails to Transcode Opus

[JPKribs]

Describe the bug

Since there are so many variables with Video Encoding, I don't know if this is necessarily AV1 or something in my audio but I have a few files encoded like this that always fail:

Video

Title: Movie Name (YEAR) - 1080p - AV1 - SDR Codec: AV1 AVC: No Profile: Main Level: 8 Resolution: 1920x960 Aspect ratio: 2:1 Anamorphic: No Interlaced: No Framerate: 24 Bitrate: 1298 kbps Bit depth: 10 bit Video range: SDR Video range type: SDR Color space: bt709 Color transfer: bt709 Color primaries: bt709 Pixel format: yuv420p10le Ref frames: 1

Audio

Title: French - OPUS - 5.1 - Default Language: fre Codec: OPUS AVC: No Layout: 5.1 Channels: 6 ch Bitrate: 214 kbps Sample rate: 48000 Hz Default: Yes Forced: No External: No

Subtitles - Not Selected

Title: English - SUBRIP Language: eng Codec: SUBRIP AVC: No Default: No Forced: No External: No

Summary

Swiftfin does not attempt to transcode this AV1 file down and I get the following screen:

Application version

1.1.1 & Main

Where did you install the app from?

App Store

Device information

iPhone 13, iPhone 15 Pro, & iPhone XS

OS version

iOS 17.6

Jellyfin server version

10.9.9

[JPKribs]

I realized using the https://github.com/jellyfin/Swiftfin/pull/1169 custom profile, I can force it to transcode but it tries to retain Opus and fails to play.

When I had this set up with the decoder check, Opus is removed from both the DirectPlay and Transcode profiles, resulting in this working:

[holow29]

Opus 2 channel works for me on Native player, even with AV1 files (which transcode video but leave audio). 5.1 channel is not working, so I assume only stereo works. I will need to see if device profile can be made that granular; I am hopeful using CodecProfiles can do this.

Can't test this at the moment, but this is something I hastily threw together: https://github.com/jellyfin/Swiftfin/compare/main...holow29:Swiftfin:patch-2

[JPKribs]

Just gave it a shot and unfortunately I'm still seeing the same issue. AV1 transcodes to H.264 x Opus 5.1 which fails on Native.

I'm not an expert on codecs. Is there any benefit to using Opus 5.1 to Opus 2.0 vs Opus 5.1 to AAC/AC3/etc?

[holow29]

Appreciate you testing it! Ideally, we want to allow Opus 2.0 to passthrough if available. In order to do so, I think it needs to be in the TranscodeProfile. This is my understanding of how it works: an AV1 file with Opus 2.0 will look at DirectPlay profiles -> no AV1 so need to transcode video -> look at TranscodeProfile -> Opus allowed so mux and passthrough. Unfortunately, I don't know where the CodecProfile comes into play. Right now, it shouldn't be doing Opus 5.1 to Opus 2.0; it is likely just passing through Opus 5.1 still.

There are a few possibilities I can think of:

• Since I don't know enough about CodecProfiles, I could have crafted this one incorrectly.
• CodecProfiles are not checked for TranscodingProfile (would need to be fixed server-side I think)
• CodecProfiles are broken server-side (just putting it out there because tht is always a possibility...)

To rule out the first possibility, I would like to test with a mp4 container with hevc or h264 video and opus 2.0 audio to see if it still DirectPlays or if the CodecProfile forces DirectStream. I don't have such a file on-hand and unfortunately my dev setup is not very usable atm. It shouldn't be hard to craft such a file using ffmpeg compiled with libopus though.

Update: I am now also seeing that iOS 17/tvOS 17 might be required for any Opus support in Native player. This would be the first time we are running into a version difference. Technically Swiftfin supports iOS 15+ right now, so this would need to be dealt with if true. Easy to add a check in, though.

[JPKribs]

Update: I am now also seeing that iOS 17/tvOS 17 might be required for any Opus support in Native player. This would be the first time we are running into a version difference. Technically Swiftfin supports iOS 15+ right now, so this would need to be dealt with if true. Easy to add a check in, though.

Interesting. And that's the targeted build version correct? NOT the version of the system running it?

If so, I believe we can add build flags around Opus in the device profile and transcoding profile.

[holow29]

Based solely on an Apple Employee: https://forums.developer.apple.com/forums/thread/759210 However, I haven't tested it myself, and it didn't seem definitive. It is easy enough to add flags, as you said, but that doesn't solve the issue with Opus 5.1 unfortunately as you know. I merely mentioned as a note to investigate at some point.

[JPKribs]

@holow29 With how https://github.com/jellyfin/Swiftfin/pull/1169 has developed, how do you feel about removing Opus from the Transcode & DirectPlay profiles for the Native Player?

My thinking is that the 5.1 Channel causes headaches and we don't appear to have a good way to distinguish between 2.0 and 5.1 at this time. Removing Opus from Native, then people can add it back in if they know what they're getting themselves into using the changes in https://github.com/jellyfin/Swiftfin/pull/1169