Support for mpris interface

[petrmanek]

I'm using QtWebFlix regularly now and I have a feature suggestion. It'd be awesome if QtWebFlix could support the mpris D-Bus interface.

This would allow:

• to play/pause video playback using tools like playerctl and media keys on the keyboard,
• to rewind/fast forward video playback,
• to adjust video volume,
• to report currently playing video name in monitor bars.

[omni6]

Would this give the ability to control with KDE-connect?

[petrmanek]

@omni6 I think yes. As far as I found, KDE supports mpris, and KDE-connect uses it to control multimedia playback.

[gort818]

@petrmanek I would love to, but I have no idea how to do it at this moment. Any help would be appreciated.

[petrmanek]

@gort818 I'd be happy to give you a hand. I just need to get a bit more familiar with the code base.

[petrmanek]

I found this adaptor for bridging mpris and Qt. Seems like the toughest part is going to be figuring out whether playback is going on and controlling it via JavaScript.

[petrmanek]

By quick examination of mainwindow.cpp and some fiddling around in my browser's dev console, I have made the following observations:

• document.querySelector('video') can still be used access the HTML's <video> object.
• The property paused can be used to check if the video is playing or not. If everything else fails, checking it once per second using e.g. QTimer can work nicely to determine the player's playback state.
• If used properly, a JavaScript listener could be added to the <video> to notify the Qt program that player's playback state has changed. The only thing left to figure out is how to asynchronously signal a QWebView from JavaScript.
• To play/pause the video, the functions .play() and .pause() can be used without problems.
• To adjust volume, the property volume can be used with a float from 0 to 1. Netflix's UI seems to adjust automatically even when the volume is set from the console.
• I don't know how to get the current playing video's name. This picture shows the DOM hierarchy, from which it may be extracted: https://gyazo.com/2f11b18ce79c05ef153d83073518c762

[petrmanek]

So, to showcase the referenced features I have created a simple fork of the repo. It can issue basic commands to the video player but still has no ties to mpris.

At the moment, I'm a bit hesitant about adding dependency on the D-Bus adapter I found, as it doesn't seem to be present as a package in any of the major Linux distributions. Does anyone have a better alternative?

[omni6]

I can package it for arch, looks quite easy...

[omni6]

It will become a split package... we would need qtmpris and qtdbusextended. So far no problem but atm a bit out of time... Will do this evening.

[omni6]

Ok, both packages build with no problems. Can immediately upload... you need it for development? Otherwise i would wait as long we need it as dependency for qtwebflix...

[petrmanek]

@omni6 Cool, feel free to put them up to AUR.

[omni6]

https://aur.archlinux.org/packages/qt-dbus-extended-git/ https://aur.archlinux.org/packages/qt-mpris-git/

[petrmanek]

Check out this commit. It can toggle play/pause and full screen and report back player state playing/paused/stopped. I have tested it and it seems to work nicely with playerctl.

[omni6]

Check out this commit. It can toggle play/pause and full screen and report back player state playing/paused/stopped. I have tested it and it seems to work nicely with playerctl.

Basic control from kdeconnect works, i can play/pause/resume. But it would be great if you can get the dead buttons to work. Don't know if that is possibe but instead of skip we could use that button for fast forward/backward. Picture of now playing movie would also be nice.

[petrmanek]

@omni6 Check it out, I've put up another iteration. This one should start animating the time bar in such a way that it's possible to see video duration and the current time position within the video.

Unfortunately, I've encountered difficulties when trying to set the time position within the video to absolute or offset values. Seems like Netflix has some sort of protective scripts which determine whether the position jumps suspiciously large intervals.

[gort818]

@petrmanek This is awesome! Unfortunately I do not know much about mpris, how would we support users of other distros? Could we make mpris a separate class, move it out from mainwindow? I think I know how to get the name of the video playing in netflix.

[gort818]

@petrmanek I was able to get the text of the video title like this,

elem = document.querySelector('.PlayerControls--control-element.video-title .ellipsize-text '); elem.innerHTML elem.textContent

Hope that helps!

[petrmanek]

@gort818 Glad you like it! 🎉

mpris itself is built on D-Bus, so most of the modern SystemD-powered distros are supported out of the box. In addition, popular WMs like KDE, Gnome, xfce or mate have mpris-capable widgets or sidebars, making it a really convenient feature.

Still, qtmpris and qtdbusextended can now be considered to be dependencies (or we can put some macro magic around them to make them optional).

For this reason, they either:

1. need to be either included as a part of QtWebFlix's package, or
2. need to be packaged on their own (like @omni6 did for arch's AUR) and be referenced, so that package managers will install them prior to QtWebFlix.

With regards to the code, the current state is rather experimental and temporary. It's definitely a good idea to move the mpris-specific additions out into a separate class / file. It'll however be necessary to expose mainwindow's web view, since the new class will want to run JavaScript commands on it.

Thanks for the suggestion concerning the title. I have already put it to use (see the latest commit). It seems to work nicely for movies. For TV shows it, however, concatenates the show's title with the episode name, like so:

MythBustersS4:E15Steam Cannon

[gort818]

Nothing a lil regex magic cannot fix, give me a minute

[gort818]

elem.innerHTML.replace(/(<([^>]+)>)/g, " ").replace(/ +(?= )/g,'').replace(/ +(?= )/g,'').trim();

[petrmanek]

@gort818 Nice, seems to work.

[gort818]

@omni6

Picture of now playing movie would also be nice.

I will try and see if I can get the thumbnail, no guarantees but that would be awesome.

[petrmanek]

@gort818 If you can get me a public URL of the thumbnail, I already have a mpris metadata key for it. ;)

[gort818]

@petrmanek I cannot seem to get it, it seems the box art images are from a http request.. I do not know how to grab the response data.

here is the header

Host: occ-0-33-37.1.nflxso.net
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.netflix.com/watch/80179786?trackId=14170035&tctx=1%2C1%2C6128ba7e-b24d-43df-adf5-50dee97b9ee3-2897852%2C1ef4d6c9-aa5a-46fb-9df5-e72802422ec4_51598400X19XX1546203694257%2C1ef4d6c9-aa5a-46fb-9df5-e72802422ec4_ROOT
Connection: keep-alive

and here is the GET request

https://occ-0-33-37.1.nflxso.net/art/8a79c/98ef726068a6f753c6ae8c4f2507717af028a79c.jpg

[gort818]

@petrmanek I have noticed many other qt programs that use mpris without extra dependencies.

Here how it was implemented in babe https://github.com/KDE/babe/blob/master/src/kde/mpris2.h

[gort818]

I created a new branch with qtmpris and qtdbusextended incuded

https://github.com/gort818/qtwebflix/tree/mpris

it will build the libs that way we dont have to worry about packaging it

Edit. NVM just going to make it more complicated

[petrmanek]

@gort818 Ad thumbnail: We don't need the actual data from the request. Just the URL is fine for mpris (it has a dedicated ArtURL metadata key). Can you retrieve that somehow? Also, I've noticed that the <video> object has a poster property which is supposed to contain a similar URL, perhaps that might help.

Ad KDE/babe: That would be my preference as well. Unfortunately, the implementation seems rather minimalistic. I've also found a fork of this repo, which attempted to support mpris. However, according to issues and the commit log, it seems like they didn't get too far implementation-wise. If you have any other suggestions, feel free to reference them -- I haven't found too many thus far.

Ad branch: Kudos to you if you actually got the project to build. I've attempted the same thing but rather unsuccessfully. Since they're both on GitHub, I'd definitely suggest including both libs as submodules instead of copy-pasting them into the repo. This way, we can keep up with any modifications in the future. I'd also prefer linking both libs statically to avoid any clashes in /usr/lib in distros (like arch), where you can install them independentely of QtWebFlix.

At the moment, I'm a bit at loss about where to continue the development of this feature. Should I give you permission to push to my fork? Would you rather prefer a pull request to the mpris branch? Or should I disregard the new branch for the moment and continue working on the fork, which we'll merge at a later time?

[gort818]

thumbnail is a pain...

item= document.querySelector('video').offsetParent.id // gets us the title id
curl -i -H "Accept: application/json"  "https://www.netflix.com/title/80131551"  // get us json info
grep -Po 'image":.*?[^\\]",' text.txt   // gets us the image 

Hope that helps! now just to add it all to qt :)

I would say continue working on the fork, you can see use the mpris fork for reference to see how I built the libs and the executable if we go that route.

Thanks so much for your efforts I really like what you have done it is awesome!

[gort818]

Clementime player uses mpris you can check that one out.

also I see ubuntu has a package https://launchpad.net/ubuntu/+source/qtmpris/0.1.0-2

[gort818]

here is an idea to implement it...

 // get netflix title
   view->page()->runJavaScript(
        "document.querySelector('video').offsetParent.id",
        [](const QVariant &result){
            qDebug() << "Value is: " << result.toString() << endl;
        }
    );

   // using the title add it to 'https://www.netflix.com/title
    QProcess* proc = new QProcess();
    QString cmd( "/bin/sh" );
    QStringList args;
    args << "-c" <<"curl -s -i -H 'Accept: application/json'  'https://www.netflix.com/title/80131551' | grep -Po '\"image\": *\\K\"[^\"]*\"' ";
    proc->start(cmd, args);
    proc->QProcess::waitForFinished(-1);
    QString output(proc->readAll());
    output.remove(QRegExp("[\\n\\t\\r]"));
    output.remove(QRegExp("[^a-zA-Z:/.-\\d\\s]"));
    qInfo()<< output;

[petrmanek]

@gort818 Check it out! I've devised a pure Qt way of doing the same thing without having to launch another process and rely on curl and grep being installed.

The metadata now looks like this:

$ playerctl metadata
{
  'mpris:artUrl': <'https://occ-0-3141-2773.1.nflxso.net/art/4c566/9f9ff80dbad539bfced0e018edffe601a594c566.jpg'>, 
  'mpris:length': <int64 3016192000>, 
  'mpris:trackid': <objectpath '/com/netflix/title/70141922'>, 
  'xesam:title': <'MythBusters S4:E28 Anti-Gravity Device'>
}

[gort818]

@petrmanek That is beautiful! Nicely done! Thanks so much! We should have a check if url starts with netflix then run this since it is specific to netflix.

@omni6 check it out!

[gort818]

Awesome looking good!

[petrmanek]

Sweet!

[gort818]

Here is how to get amazon prime video title

document.querySelector('div.webPlayer').innerText.split('X')[0].trim()

[omni6]

Can arrow keys, enter and back get implemented?

[petrmanek]

@omni6 It all comes down to bypassing Netflix's protections. If you try to modify <video>'s position directly, it just redirects you to that error page.

Perhaps if instead we tried to simulate a click on the position bar, we could achieve the result without raising suspicions.

@gort818 Do you think we could find the position bar in the DOM tree?

[omni6]

I don't mean to modify the movie Position. This is meant to navigate on the streaming webpage.

[petrmanek]

I guess we could make prev/next navigate TV show episodes. However, I'm not sure how to get to their IDs (or URLs). Any suggestions?

[omni6]

Oh, i was sure about that it is possible to navigate with keyboard on the webpage. You know, with the arrow keys, enter and maybe escape/backspace for back and send this commands over mpris... but navigate with keyboard doesn't work.

[gort818]

Here is how to get box art for amazon prime

document.querySelector('div.av-fallback-packshot').childNodes[0].getAttribute('src')

document.querySelector('div.av-bgimg__div').getAttribute('style')

It is one or the other.

I will try to find position bar for netflix

[gort818]

@petrmanek try this

document.querySelector('div.PlayerControlsNeo__progress-control-row.PlayerControlsNeo__progress-control-row--row-standard')

No idea how to interact though

[gort818]

@petrmanek I got fast forward and backword... no seek yet :(

// fast forward
document.querySelector('button.touchable.PlayerControls--control-element.nfp-button-control.default-control-button.button-nfplayerFastForward').click()

//rewind
document.querySelector('button.touchable.PlayerControls--control-element.nfp-button-control.default-control-button.button-nfplayerBackTen').click()

[gort818]

@petrmanek and here is next epsidoe

document.querySelector('button.touchable.PlayerControls--control-element.nfp-button-control.default-control-button.button-nfplayerNextEpisode').click()

[gort818]

@petrmanek and here is the seek bar

document.querySelector('div.scrubber-bar')

Cannot find a way to intereact

[gort818]

@petrmanek and here is the back button maybe use it as stop ?

document.querySelector('button.touchable.PlayerControls--control-element.nfp-button-control.default-control-button.button-nfplayerBack.tooltip-button.tooltip-button-pos-center.tooltip-button-align-right').click()

[gort818]

For specific netflix stuff we can do something like this.

QString MprisInterface::getSite() {
    QString site = webview->url().toString();
    if ( site.contains("netflix")) {
       QString = "javascript code"
        return code
}

and then call it

void MprisInterface::getMetadata(std::function<void(qlonglong, const QString&, const QString&)> callback) {

    QString code = getSite();
              webview->page()->runJavaScript(code, [callback](const QVariant& result)

[petrmanek]

@gort818 Wow, you've been busy! I'm going to put the code to a good use.

I was thinking about supporting multiple different services other than Netflix (like Amazon Prime, etc.), and I think there's a better solution than to basically have series of if's or switch's in every function.

Instead, I would propose to solve this by inheritance. Turn MprisInterface into an abstract interface, and then have the actual implementation present in NetflixMprisInterface and PrimeMprisInterface where it's safe to assume we're dealing with the single respective streaming service. This way we may also avoid branching the mpris logic the same way in every single function body.

The job of MainWindow would then be to communicate with MprisInterface and occasionally exchange it for other implementation when user switches services. What do you think about that?

P.S. I don't have Prime subscription, so you'll have to help me test those features.

[gort818]

@petrmanek Yeah that sounds good if we are going to support other services, I only have a netflix and prime account right now, so I guess when can start with those.

Also I thought about what you said earlier I will add the qt-mpris libs as submodules and compile them as static libraries I think that is the easiest solution, I will let you know when I have pushed to the mpris branch.

Again thanks for all your hard work I am loving it !

[gort818]

@petrmanek Ok got everything building with static libs https://github.com/gort818/qtwebflix/tree/mpris

I believe you just have to include the header files and you should be good to go!