search

microsoft / Windows-Dev-Performance

A repo for developers on Windows to file issues that impede their productivity, efficiency, and efficacy
MIT License
438 stars 21 forks source link

Transcoder throws exception when using HLS or DASH as source #26

Closed theothernt closed 4 years ago

theothernt commented 4 years ago

Environment

Item Value
OS, Version / Build 10.0.19041.0
Processor Architecture AMD64
Processor Type & Model Core i7
Memory 12GB
Storage Type, free / capacity SSD 512GB
Relevant apps installed VS 2019 16.6.5 + UWP SDK

Description

I'm trying to create a video preview of a live video stream (eg. 4 seconds of a HLS or DASH stream). The MediaTranscoder API has a method which looks like it should work with an AdaptiveMediaSource (ie. streaming video source) but does not - instead, I get an exception.

Steps to reproduce

  1. Find a HLS or DASH stream (there are many test feeds out there, not encrypted)
  2. Create an AdaptiveMediaSource from the URI
  3. Pass the source to Transcoder.PrepareMediaStreamSourceTranscodeAsync

Expected behavior

I expect the transcode to complete resulting in an IRandomAccessStream which I can then view directly or save to disk.

Actual behavior

When using a source from AdaptiveMediaSource.CreateFromUriAsync I get an InvalidCastException from the transcoder.

If I use MediaSource.CreateFromAdaptiveMediaSource to convert the source type, I get a NullReferenceException.

App to reproduce the behaviour... https://github.com/theothernt/TranscodeStreamIssue

asklar commented 4 years ago

@theothernt Thanks for putting together a repro! I don't work in media foundation but this seemed like an interesting problem so I figured I'd dig in to see if I can glean what might be going wrong. I see the InvalidCastException too, which seems to be coming from the async library that MF uses to do transcoding. Jumping back in time between all the different threads where the original error happens, I see a failure in MFTranscode.dll - when trying to QueryInterface the AdaptiveMediaSource onto IMFMediaSource. AdaptiveMediaSource seems to implement IMediaSource, but not IMFMediaSource. Someone from MF would need to comment on whether this is intended or something that was missed.

asklar commented 4 years ago

sent mail to MF, stay tuned

theothernt commented 4 years ago

Thanks for looking into the issue @asklar!

asklar commented 4 years ago

Hey @theothernt, I got a response from the MF folks:

The AdaptiveMediaSource does not implement a MF media source. Instead it uses Media Source Extensions (MSE) to feed data into the MF pipeline. MSE implements the MF media source. We do not currently have a public way of accessing the IMFMediaSource that is used when working with the AdaptiveMediaSource. Also, the AdaptiveMediaSource is tuned for real-time playback. Trying to get faster than real-time for use with the transcode APIs would likely result in the AdaptiveMediaSource switching to a low quality stream. Part of the adaptation logic uses the playback clock position as well. It would be interesting to understand more about the scenario they are trying to achieve. I would think that transcoding should be done using a fixed quality stream and not something that adapts to network conditions.

So it sounds like the scenario/usage pattern that the app uses isn't something that's supported. Though I agree that the exception being "invalid cast" is kind of non-sensical if one doesn't know what's happening behind the scenes. Hope this helps!

theothernt commented 4 years ago

Thanks, I thought this might be the case. I was looking into this purely as an experiment as I've worked with broadcast video platforms before. But after research, it's far more efficient to generate video stream previews on the server. For example...

  1. Netflix simply streams a trailer or play the movie/show when you hover over an item.
  2. YouTube loads a pre-made animated WEBP when you hover over a video.