Audio tracks and Timed meta data tracks are 0 in uwp mediaplaybackitem

[touseefbsb]

I am using this library in my uwp project. And I am using MediaPlayerElement to play the media, just like in sample of this repo, the only thing I changed is I amusing MediaPlaybackItem to set the source instead of MediaSource directly

var source = MediaSource.CreateFromMediaStreamSource(mss); var item = new MediaPlaybackItem(source); ViewModel.PlaybackSource = item;

this media playback item object should have audio tracks for this specific file, and also closed caption tracks represented as TimedMetaDataTracks. But using break point I know the count of them is 0. Meanwhile if I play that file with Movies and TV default app of windows 10 they show a button to change the subtitles and to change audio tracks for that file, I wonder why isnt my media player element showing that button? and why is it 0 when I use the breakpoint?

Thanks in advance.

[brabebhin]

Hello

This is related to #195

[lukasf]

There are multiple issues here.

First, FFmpegInterop does not support subtitle tracks. MediaStreamSource has no way to add subtitles, it only supports audio and video tracks. So we cannot support that in a simple way. This is very unfortunate, and makes it difficul to use the lib for a full featured video player. Please vote on my UserVoice item if you want this to be possible:

https://wpdev.uservoice.com/forums/110705-universal-windows-platform/suggestions/31792825-add-timedtextstreamdescriptor-to-enable-subtitles

Second, FFmpegInterop currently only supports one audio track. So even if the file contains multiple languages, FFmpegInterop will only use the first (or default) track. It would be possible to change FFmpegInterop to support multiple audio tracks. But it requires bigger rework of parts of the library, and this has not been done yet. MediaPlayerElement will automatically show the track selection button if more than one track are provided in the MSS, but currently only one track is provided. See #176

Third, the MediaPlaybackItem will only populate it's tracks when the file is actually opened. You'd need to register to its AudioTracksChanged event, then open the file. When the file is loaded in MediaPlayerElement and event is fired, you will find information about the tracks. But as I said, you'd currently only find one track in there.

[touseefbsb]

@mcosmin222 What is the time expected for that pull request to be merged and does that fix this issue of subtitles and audio tracks ? thanks

[lukasf]

@touseefbsb I am pretty sure that it will not solve any of those problems.

[brabebhin]

You can implement subtitles using an overlay textblock and custom parsers. This is how i did it in my app.

[brabebhin]

@lukasf , it is somewhat related because you still cannot use any features of the MediaPlaybackItem aside from starting at position 0.

[touseefbsb]

@lukasf thanks for clarificatiion but until we can support multiple audio tracks and subtitles there is no use to register to Autotracks changed event. So this library really needs to support it somehow. because it is a major feature requirement.

[touseefbsb]

@mcosmin222 we cannot use any features listen by mediaplaybackitem? well that is very unfortunate because it surely has lot of features, does that mean we cannot use mediaplaybacklist as well?

[touseefbsb]

@mcosmin222 also can you please further clarify on the subtitles? how can we design custom parsing of the subtitles? I mean how can we extract it from the file and then show it on the overlay textblock? I get the idea tht we can make a overlay textblock with custom media transport controls but I dont know much abt how to get the subtitles from the file.

[brabebhin]

mediaPlaybackList is a different class. Sure you can use that.

But, the current state of the library, for MediaPlaybackItem only:

You cannot use audio tracks which have start position other than 0. This will be fixed when #195 is fixed. You cannot switch between multiple AudioTracks or VideoTracks. (these are pretty hard to get by to test around, since I do not have any such files) You cannot use native subtitle rendering.

What you can do is extract the subtitle and display it some other way. But this isn't really simple.

Perhaps there is a way to do this through the media stream source itself, probably by merging the subtitle directly into the video sample. But I wouldn't hazzard a guess on this one.

[touseefbsb]

@mcosmin222 when #195 is fixed can we hope that mediaplaybackitem features can be used ?

[brabebhin]

Not all of them, no. We have to support each individual feature separatly. I am not sure if you noticed, but MediaStreamSource isn't an API that's very high on the priority list of the media guys.

Anyway, if you want a fully featured media player, you will have to get your hands dirty. MediaStreamSource doesn't support a lot of scenarios, and you will not be able to build everything just with this library. You need others as well.

[touseefbsb]

actually if we dont use the library then, we can implement all the features with mediaplaybackitem but then we cant use all the codecs, this library was supposed to get us the codecs, and now it is breaking the basic features in the platform :/

[brabebhin]

Welcome to the party. You have to build features on top of this one to complement what is missing.

I long suspected that if we were somehow able to expose this MediaStreamSource (or FFMPEG itself) as a Stream (by implementing the IRandomAccessStreamInterface), we could technically get all the features of MediaPlaybackItem (and probably more). But that is one dream of mine that I stopped chasing 2 years ago. :)

Specifically since we need a lot of docs on how MPI works. I doubt Microsoft will ever provide these.

[touseefbsb]

@mcosmin222 I know its is not a sure thing Microsoft will make it but it is still worth a try. I made this uservoice a while ago and it looks like it still hasnt gained the deserved attention, if you can help it spread along maybe they will consider it. https://wpdev.uservoice.com/forums/110705-universal-windows-platform/suggestions/19248226-codecs-for-uwp-media-element

[lukasf]

For adding subtitles: It is possible to add external subtitle sources into the pipeline, and have MediaPlayerElement render them automatically (and also provide subtitle track selection). I did some testing with that already. It basically worked and also allows to add subtitle entries "on-the-fly" during playback. But I also had problems, especially with formatting (you do not want the subtitle on top left corner).

https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/media-playback-with-mediasource#add-external-timed-text-with-timedtextsource

Maybe it would be possible to raise events in FFmpegInterop whenever subtitles are to be shown (ffmpeg does support subtitle tracks). Then we could add the decoded subtitles from ffmpeg as TimedTextCue to the TimedMetadataTrack and get them shown. The system also supports bitmap based subtitles.

There is also TimedTextSource which is a lot better, but it only supports extracted subtitle tracks (like .srt files and others). It does not work with the subtitles embedded in a video file.

[brabebhin]

The problem with timed text source is that it doesn't allow you too much freedom. For example, it doesn't support presentation timestamp adjustments. I ended up implementing the exact same API myself, and I got more satisfying results.

[lukasf]

Yes TimedTextSource is not what we need. I was primarily talking about TimedMetadataTrack, and there you add each cue entry manually with a start and end time. Still, rendering is done by MediaPlayerElement. But no matter which way you choose, it is quite a lot of manual work in the app code to get this running. It is unfortunately not something that we can just add into the lib and it will automatically work. For that, we would need subtitle support directly in MediaStreamSource.

[brabebhin]

As I said, I did this "manually", but on C# level, using custom media transport controls. It is somewhat easy to add the Textblocks (in my case it was a ListView of Textblocks), disable the scrolling, disable the animation, and use a simple enumerator to figure out which subtitles to display.

The biggest challenge by far was finding the parsers, but there is a C# .net parser library (which uses forbidden APIs). You just need to adjust that one to winRT.

[lukasf]

Guys, look what I just found in latest SDK Preview:

namespace Windows.Media.Core {
  public sealed class TimedMetadataStreamDescriptor : IMediaStreamDescriptor, IMediaStreamDescriptor2
}

Microsoft heard us, yay!! 😀

[brabebhin]

Nice. :) a little unexpected though.

[lukasf]

Indeed unexpected. They probably already had it on their list. Anyways, good news for us!

[touseefbsb]

@lukasf Does this mean now we can use all MediaPlaybackItem properties including audio tracks and embeeded subtitles? :)

[brabebhin]

Just embedded subs.

[touseefbsb]

well that is halfway through, though a great relief, but we have to wait because its still preview sdk and when it releases in march it will come with next major release of windows 10 I guess and we can only use it if our app targets that sdk and user is using latest version as well ( i.e : next version to fall creators update )

[lukasf]

No, it just means that with next major Windows update, it will be theoretically possible to output subtitles through MSS. It will require us quite some work to add support for multiple audio streams, plus multiple subtitle streams into FFmpegInterop. Both requires many changes. But once finished, both should work out-of-the-box in an app, without any custom subtitle parsing or custom UI or whatever.

[lukasf]

Multiple audio streams would work on older Windows versions as well. But subtitles would only work for users running Windows VNext.

[touseefbsb]

alright then lets keep synced on this issue and lets make those features after the next sdk is released, specially the audio streams, sounds good? @lukasf

[lukasf]

We could already add multiple audio streams now. It does not depend on next SDK. Only subtitles depend on next SDK. It is only a question of who wants to do the work ;)

Plus, right now we already have 8 open PRs. I think @khouzam is the only one who can merge PRs, which is kind of a bottleneck. It looks like he currently does not have enough time to look through all of the RPs and merge them. This makes progress of the library a bit slow right now.

Actually, I would like to work on multi stream stuff. But I already have 4 PRs open, plus I have some more things prepared. I don't want to start more work if I am not confident that it will end up in the library in a reasonable time. Also, adding multiple streams would require major rework. It would be a bad idea to start that when so many PRs are open (this could cause many merge problems).

[brabebhin]

We also need a pool of test files to ensure there are no hidden regressions.

[lukasf]

That would be very helpful indeed.

[lukasf]

I think it might also be a good idea to introduce a "release" branch, once all current PRs are merged and tested. Then we could merge new PRs a bit more quickly into master and test the complete set of changes. Once we have reached a good and stable state, we can bring that to "release" again. Similar to how other projects create a nuget package at a specific point - just that we cannot really distribute compiled binaries of ffmpeg, so we have to use a branch instead.

Currently, we'd have to be sure that our PRs already have "release quality" before even merging them. That is difficult and can cause a lot of open, long running PRs, and lead to merge conflicts.

Not saying that we should rush PRs into the repo, of course. Master should always have "good quality", so PRs should be properly tested and reviewed. But a release branch would allow us to consolidate a set of changes and test the complete set, before releasing it...

[touseefbsb]

@lukasf yes u r right there must be a production branch ( master ) and a dev branch, and on dev branch we can test new features before releasing and after it is carefuly reviewed for bugs then we can push it to the production branch

[khouzam]

Absolutely, I do agree that we need to add some more organization to the repo, create proper branch structures, version tagging, increase the test coverage, add automated builds... Especially with the current number of changes that you are all investing, I want to make sure that we can keep the quality and have developer know which code to take and feel confident of the quality.

[touseefbsb]

@lukasf does this mean we cannot use external file subtitles ( like srt files ) either? until the next sdk?

[lukasf]

@touseefbsb Extrenal subtitles are already supported through TimedTextSource, since beginning of Windows 10. But passing subtitles that are embedded in a video file through MSS will only come with vNext.

[touseefbsb]

alright thanks :)