Proposal: System Media Transport Controls for desktop apps

[JaiganeshKumaran]

Proposal: System Transport Controls for unpackaged apps

UWP apps that use the built-in media controls are automatically integrated System Media Transport Controls. This appears when you press the volume up or volume down key on your keyboard (if you have one) or when locked it shows on the lock screen with the option to pause, go forward and background however UWP apps that don't use the built-in controls can also be integrated.

Summary

Currently, UWP apps can manually integrate themselves using the APIs in Windows.Media namespace. Desktop apps on the other hand can't get integrated very easily especially non-packaged apps. Packaged desktop apps can technically work but because of the dependency of CoreWindow it's not that straightforward. Some Microsoft apps like the new Microsoft Edge do support but it requires some workarounds.

This proposal aims to remove the dependency of CoreWindow for System Media Transport Controls and to provide a simple and straightforward API that will work the same across Win32 and UWP without any change. Currently Win32 apps need to IInitializeWithWindow to access the System Media Transport Control APIs but I think that's not really required.

Rationale

System Media Transport Controls makes it extremely easy to pause background media. Also, Windows automatically pauses media that's already playing when you start playing something else or when you disconnect your speaker.

• System Media Transport Controls provide a better media experience for users and today many Win32 media don't support this feature yet.

Scope

Capability	Priority
This proposal will allow developers to integrate any app with the System Media Transport Controls without any differences or workarounds for Win32 apps	Must

Important Notes

Since this feature was first introduced with Windows 8, these APIs should work on these platforms as well.

[jonwis]

Great proposal! Would it be reasonable to define a method like:

namespace Microsoft.ProjectReunion.Media.SystemMediaTransportControls 
{
    static Windows.Media.SystemMediaTransportControls GetForWindowId(WindowId window);
}

... where you could get a WindowId from an HWND, CoreWindow, AppWindow, etc? That is, you get the real object, but we handle all the "is this a corewindow or an hwnd? should we call ISystemMediaTransportControlsInterop?" stuff under the covers?

[JaiganeshKumaran]

Great proposal! Would it be reasonable to define a method like:

namespace Microsoft.ProjectReunion.Media.SystemMediaTransportControls 
{
    static Windows.Media.SystemMediaTransportControls GetForWindowId(WindowId window);
}

... where you could get a WindowId from an HWND, CoreWindow, AppWindow, etc? That is, you get the real object, but we handle all the "is this a corewindow or an hwnd? should we call ISystemMediaTransportControlsInterop?" stuff under the covers?

Yes, we can do that.

[mdtauk]

Could there be support added for a Keyboard Shortcut or a Taskbar Context Menu entry, to bring the control on screen?

I know taskbar icons can include limited media buttons in the app preview flyout - but before I got my current keyboard (with a volume wheel), I had no way to adjust the volume, other than from the volume mixer flyout - and that does not include the media preview, details or controls.

[JaiganeshKumaran]

@mdtauk There are rumors that Microsoft is going to put the controls in the volume flyout of the system tray but we don't know for sure.

[mdtauk]

@mdtauk There are rumors that Microsoft is going to put the controls in the volume flyout of the system tray but we don't know for sure.

I imagine this would replace the current Media Transport controls

[mveril]

Great proposal! Would it be reasonable to define a method like:

namespace Microsoft.ProjectReunion.Media.SystemMediaTransportControls 
{
    static Windows.Media.SystemMediaTransportControls GetForWindowId(WindowId window);
}

... where you could get a WindowId from an HWND, CoreWindow, AppWindow, etc? That is, you get the real object, but we handle all the "is this a corewindow or an hwnd? should we call ISystemMediaTransportControlsInterop?" stuff under the covers?

A similar C# wrapper can be found here on the WindowsInteropWrappers repo. Big thanks to the author @AdamBraden. It's should be transposable to other wrappers of the WindowsInteropWrappers repo.

[JaiganeshKumaran]

Could there be support added for a Keyboard Shortcut or a Taskbar Context Menu entry, to bring the control on screen?

I know taskbar icons can include limited media buttons in the app preview flyout - but before I got my current keyboard (with a volume wheel), I had no way to adjust the volume, other than from the volume mixer flyout - and that does not include the media preview, details or controls.

If you have a Bluetooth headphone/headset, you may be able to access the Media Transport Controls by pressing the volume up or down key. However this only support by a few devices.

[jonwis]

Hey @AdamBraden - want to rework your samples into C++/WinRT and add a feature proposal / API spec for Project Reunion? :)

[AdamBraden]

I think the work @mikebattista has planned for the SDK should take care of this.

[ePirat]

What is the state of this? I always wanted to add support for this to VLC, but it seems with the currently available APIs this is impossible.

[AdamBraden]

@ePirat - have you had a chance to look at ISystemMediaTransportControlsInterop::GetForWindow? https://docs.microsoft.com/en-us/windows/win32/api/systemmediatransportcontrolsinterop/

[ePirat]

@AdamBraden Thanks, although checking the docs it is not entirely clear to me how this API is supposed to be used. What is the riid I need to pass? (And it seems the description for the supposedly out argument mediaTransportControl is wrong as it is identical to the one for appWindow?)

[mveril]

Hello @ePirat Maybe you can find the solution by looking at how this API is used in the source code of the chromium project.

[ePirat]

@mveril Thanks, unfortunately that looks not trivial to use. In addition it does not seem possible to use in VLC when cross-compiling on Linux using MinGW-w64 as it seems the relevant SDKs are not available unless building directly on Windows.

[reflectronic]

Is ISystemMediaTransportControlsInterop supposed to be usable from unpackaged apps? In Windows 11, it seems to have problems picking up on the app's name and icon.

Unpackaged app, lacking app title and icon

Packaged app, displaying app title and icon as expected

I understand that a sparse package would probably fix this, but it is a lot of cost (both in development time and $) just to have an icon appear in this menu.

[Samuel12321]

As one of the devs of ModernFlyouts, i can say this needs to be fixed urgently. The current STMC API is problematic, to the point where where @ADeltaX built a custom NowPlayingSessionManager wrapper library here to try to improve things. Aside from being a nightmare for external devs, it also causes problems for windows inbuilt media flyout as you can frequently see out of date information displayed.

The other issue with the current winrt based STMC api is that most apps dont support STMC, and the majority of apps that do support STMC only support a subset of its features.

Some of the weird quirks also need to be resolved - such as timeline only updating every 4 seconds etc...

[AdamBraden]

Is ISystemMediaTransportControlsInterop supposed to be usable from unpackaged apps? In Windows 11, it seems to have problems picking up on the app's name and icon. Unpackaged app, lacking app title and icon Packaged app, displaying app title and icon as expected

I understand that a sparse package would probably fix this, but it is a lot of cost (both in development time and $) just to have an icon appear in this menu.

As you see, it can be used, but the functionality is limited because the app does not have identity. What you are asking for a new api where you can pass along that information. However, given sparse packages as a workaround, this may not be high on the priority list.

[elibroftw]

So what is the best way to give users a media overlay / controller. I implemented SMTC but nothing shows up for my application. Should I be implement MediaControl instead?? Here is how Firefox does it:

https://searchfox.org/mozilla-central/source/widget/windows/WindowsSMTCProvider.cpp#102 https://searchfox.org/mozilla-central/source/widget/windows/WindowsSMTCProvider.h#1

[yume-chan]

To display your app's name and icon, you just need to create a shortcut to your exe in %APPDATA%\Microsoft\Windows\Start Menu\Programs. The file name and icon of the shortcut will be used, the name and icon in your exe doesn't matter.

I created mine using C++ following https://learn.microsoft.com/en-us/windows/win32/shell/enable-desktop-toast-with-appusermodelid#step-2-create-the-shortcut-and-install-it-in-the-start-screen (setting PKEY_AppUserModel_ID is not required). But create a shortcut manually in Windows Explorer also works.