search

microsoft / FFmpegInterop

This is a code sample to make it easier to use FFmpeg in Windows applications.
Apache License 2.0
1.31k stars 310 forks source link

C++/WinRT conversion of FFmpegInterop #264

Open Noemata opened 5 years ago

Noemata commented 5 years ago

Any chance Brian Beecher could finish the conversion of the cppwinrt branch to C++/WinRT throughout (some bits are still C++/CX) and then swap in the vcpkg version of ffmpeg? This would bring the lib fully into the present.

Noemata commented 5 years ago

If this is a dead project, please archive it so it's obviously abandoned.

Zofware commented 5 years ago

Try FFmpegInteropX . It's still getting updates.

Noemata commented 5 years ago

I did. I was surprised at the number of things that didn't work with that lib and the fact that perf/usability wasn't as good as this: https://github.com/ionutdanila/FFmpegInterop

I didn't get the impression the FFmpegInteropX devs are particularly keen on moving to more recent tech, which is also disappointing.

Ionutdanila's versoin seems to be the most recent working instance of FFmpegInterop I was able to find.

It's also unfortunate C++/WinRT isn't being used as much as it should be, that includes Microsoft.

It's time C++/CX went away.

Brian's code is a start on the C++/WinRT conversion, but it's largely unfinished from what I can tell :-(

I don't really have the time to muck with any of this. Looking for something I can hobble together in minutes not hours.

Given vcpkg gets you a chunk of the way there for UWP ffmpeg support in a painless way, it would be nice to see Brian hop back onboard here to finish the C++/WinRT conversion. C++/CX is painful to look at let alone work with if you're a C++ old timer. And it too is now deprecated by Microsoft, so no future in anything with C++/CX in it.

Noemata commented 5 years ago

Nice app @Zofware!

Zofware commented 5 years ago

Thanks. 😄 By the way, the main reason I switched to FFmpegInteropX is because their build script works for the latest stable version of ffmpeg (4.2.1). I'm not actually using the managed wrapper at all, which is a bit ironic I know. I just wrote a minimal C++/CX wrapper for the one feature I needed. I didn't realize CX was being deprecated so I guess I'm not the only one who needs to update to C++/WinRT. 🙄

Noemata commented 5 years ago

If you don't need their wrapper, then your life can be even easier if you use vcpkg to pull from the latest ffmpeg release. I live mostly in some UI layer, so I tend to need tooling like this. Thank you for the suggestion nonetheless. I am curious, are you streaming to an RTMP server or something else? I need to do both push and pull. The pull is working for me with the lib I mentioned; FFmpegInteropX was dying in a pecular part of the code path. Not yet sure how I'll do the push. I need hardware encoding of the outgoing video that's fairly flexible. AzureRTMPIngestLib looks very promising for the push, just that I'll need to adapt it since it targets an Azure back end.

Zofware commented 5 years ago

Thanks for the vcpkg recommendation. I build ffmpeg for UWP with none of the GPL options enabled so I'm not sure vcpkg fits my needs?

I send RTMP streams to YouTube primarily but any RTMP destination will work, in theory. I use MediaCapture to record fragmented MPEG-4 to a IRandomAccessStream implementation that redirects the bits to ffmpeg, which remuxes the fmp4 stream and sends it to the RTMP server. It's been working really well except for one thing: bitrate control. When MediaCapture uses an Intel Quick Sync encoder then CODECAPI_AVEncCommonMeanBitRate doesn't seem to work and I'm unable to thottle the encoder if the network bandwidth is less than expected. This results in the outgoing queue filling up and the stream starts to lag further and further behind real time. I'm actively trying to find a solution for this but I'm still looking.

Noemata commented 5 years ago

My understanding is that you have to specifically do something like:

.\vcpkg install ffmpeg[gpl]:x64-uwp --recurse

In order to get the GPL version with vcpkg. Don't quote me on that.

I like your approach @Zofware. AzureRTMPIngestLib seems to be a nice implementation for multi-bitrate handling. When I get a chance I might have a look at adapting it. Your approach seems straightforward also. Is any of your MediaCapture RTMP code avialable somewhere?

Noemata commented 5 years ago

One thing that doesn't get enough advertising is that the UWP version of FFmpegInterop uses Media Foundation services for certain problematic codecs in terms of licensing. So you don't have the same licensing headaches that you would have if you were to use FFmpeg out of the box on its own. A lot of what FFmpeg does aint free unless you live in France. Even with FFmpegInterop as @Zofware has identified, how FFmpeg gets built for a given project needs careful consideration. I'm not 100% certain you can say that you fall under the OS license if you are using code that isn't part of the OS even if your code is otherwise tied to the OS. Oh, and static linking to FFmpeg is a no-no from the get go. Unless you are ok with the obligation of releasing source to everything built with it.

Zofware commented 5 years ago

I only use ffmpeg to remux without re-encoding and to send via RTMP and I use a custom-built dynamically linked version with minimal options enabled so I hope I'm steering clear of licensing issues. All the video and audio encoding is done by MediaCapture in Windows.

My code is closed source but here's the gist of it:

  1. Create a MediaEncodingProfile for MediaCapture that uses a fragmented MP4 container. For the container Subtype use the MFTranscodeContainerType_FMPEG4 GUID from mfidl.h in the Windows SDK. Initialize the Audio and Video properties for AAC and h.264 respectively. I manually mimic what MediaEncodingProfile.CreateMp4() does but I change the properties as necessary to match what YouTube expects.
  2. Start the stream by calling MediaCapture.StartRecordToStreamAsync() (or use PrepareLowLagRecordToStreamAsync()) passing in the MediaEncodingProfile you created above and a custom implementation of IRandomAccessStream as described below.
  3. Implement IRandomAccessStream to take any bytes written to it and store them in a thread safe queue. I actually did this in two parts: a managed implementation of IRandomAccessStream coupled with a native wrapper around concurrent_queue.
  4. On the native side, use a worker thread to wait for bytes to arrive in the queue then use the ffmpeg library to remux to FLV and send the bytes to the RTMP server. If you Google around you can find samples showing which ffmpeg calls to make. I used bits and pieces of this and this to get started although there are probably better and more recent samples out there. Another way to figure out some of the ffmpeg calls needed is to step through the ffmpeg command-line tool in a debugger as it remuxes a FMP4 file to FLV and sends to a rtmp:// URI destination like YouTube.