pupgui2: Send Download Progress and Active Downloads to DBus

[sonic2kk]

Overview

This PR enables displaying Download Progress and Active Downloads count on Task Managers, Docks, etc that listen for the com.canonical.Unity.LauncherEntry.Update DBus signal. When downloading tools, a progress bar will now appear on the Task Manager / Dock, displaying the same progress as the progress bar on the main window, as well as a count of the total active downloads. This can be useful if you have many windows open across different screens, or conversely on a single small display like the Steam Deck where you may set a tool to download and then open another fullscreen window as screen real estate is at a premium. It provides a visual indication of download progress outside of the ProtonUp-Qt main window and download count, similar to what a browser might show when downloading files.

Despite the signal name, this is supported by KDE Plasma's built-in Task Manager widgets (it may also be supported by Latte Dock, unsure), and should be supported by GNOME as well.

If a system does not have DBus, or in Flatpak if DBus is disabled, nothing will happen. We check if we can use a given DBus bus first with isConnected().

KDE Plasma Icons-Only Task Manager

This Task Manager is also used by default on SteamOS and many KDE Plasma installations with the default panel.

KDE Plasma Task Manager

This is a built-in alternative to the Icons-Only Task Manager. I don't think it's used by default though.

Background

The tl;dr is that we have to get down-and-dirty with DBus as there is no higher-level wrapper for this functionality that I could find with Qt, at least not in Qt6. There have been requests to replace the signal with a Freedesktop Portal (https://github.com/flatpak/xdg-desktop-portal/discussions/1276) but nothing has materialized as of yet.

This started because I noticed on SteamOS that Discover displays its download progress (I don't use a Task Manager or a Dock on my main PC), and I wondered how this was possible. After much research, I discovered based on https://github.com/ValveSoftware/steam-for-linux/issues/4777 that this is done via DBus.

From this, I figured that since Qt is fairly modern, there should be a good built-in way to hook into this DBus signal, as it is standard across at least two major Desktop Environments, KDE and GNOME. There is not, and someone actually created a custom Qt C++ class called QTaskbarControl to manage exactly this in a cross-platform way, using DBus on Linux (labelled as X11, but the underlying DBus logic works on Wayland as well).

At the time, I knew absolutely nothing about DBus, so I didn't really understand how to copy this over effectively. I didn't understand when and where to use certain identifiers. Instead of working backwards from how QtDBus does it I first started with understanding how the gdbus command works and what each part of that meant. It took a long time and a lot of playing around with things in ProtonUp-Qt, but I finally got somewhere, and so here we are!

Some other things I referenced were:

• General PySide6 DBus examples: https://github.com/flyfire/pyside6-examples/tree/7de66d333cc5ec22c8a5d1c322c988692e8da72e/dbus.
• C/C++ Qt implementation of this exact signal: https://askubuntu.com/questions/65054/unity-launcher-api-for-c/65199.
• All of the PySide6 QDBusMessage docs.
• This PyQt5 gist on using QtDBus with Python: https://gist.github.com/dpineiden/8254dd35f37ee05720a0ad13abf0ad3f.

Implementation

You might know a lot more about DBus than me, and my explanations may be wrong, but I will try to explain this the way I would've liked it broken down and explained to me yesterday when I was coming at this fresh.

Emitting to com.canonical.Unity.LauncherEntry.Update via The Command Line

In order to send the DBus progress information, here's how a gdbus command would look:

# Send a 60% progress message to com.canonical.Unity.LauncherEntry.Update signal
gdbus emit --session \
    --object-path /net/davidotek/pupgui2/Update \
    --signal com.canonical.Unity.LauncherEntry.Update \
        "application://net.davidotek.pupgui2.desktop" \
        "{'progress': <0.6>, 'progress-visible': <true> }"

I'm not an expert with DBus, but here's a breakdown of my understanding:

• gdbus emit: Send a message over DBus. We specify emit because you can probably also receive messages too.
• --session: This says to use the "session bus". DBus has many busses or paths to send information. The session bus is I believe used to specify only for the current user session OR, in the case of Flatpak, only for that Flatpak session.
  • There are probably messages you'd want to send to, say, system-level processes that may not be available on the session bus. I'm not sure it's strictly necessary for this particular command to use the session bus, but it is good practice.
• --object-path /net/davidotek/pupgui2/Update - This is an identifier for the DBus message, if you use something like dbus-monitor, it allows you a readable way to see what exactly is sending this message. It doesn't have to be this name, this is just the one I chose.
  • Applications could also listen to messages on a specific path, which would make it more important, but for this example and this PR, it doesn't matter. It could just as easily be /com/valvesoftware/SteamForBSD.
• --signal com.canonical.Unity.LauncherEntry.Update - This is the signal we want to send our message to. Processes can listen for messages to a given signal, in this case, Task Managers can listen for messages on the signal com.canonical.Unity.LauncherEntry.Update and use the arguments given to that signal to do something.
  • The purpose of this signal is to update a display item on a launcher, i.e. a Dock, a Task Manager widget on a panel, etc. Or, to put it another way, to Update the LauncherEntry.
  • Technically, this is two parts:
    • com.canonical.Unity.LauncherEntry is an interface, which to my understanding is like a grouping of signals.
    • Update is the actual signal on this interface. An Update signal could exist on many interfaces.
• Everything after this is arguments to the signal and specific to the signal.
  • "application://net.davidotek.pupgui2.desktop" - Since this Updates the LauncherEntry, this argument is the .desktop file representing the Launcher Entry. To my knowledge it doesn't have to be using the application:// URI, but this was the most prevalent one I found in examples. So, we use this to tell the signal "Update applications associated to this .desktop file on any Task Managers listening for this signal".
  • "{'progress': <0.6>, 'progress-visible': <true> }". This is a dictionary that contains information that the Update signal can take and parse. You don't have to pass just these, you can actually pass more, you are basically giving a dictionary of data to the Update signal that, if it is looking for it, can parse and use. So in this case the Update signal listens for progress. You also have to set progress-visible otherwise it won't display anything (similarly, you can use this to hide the progress once the Progress is complete, as this won't happen automatically).
    • I'm not sure why the values have to be wrapped in <angle brackets>, but in my testing, they do.
    • Optionally you can add count and count-visible, like this: "{'progress': <0.6>, 'progress-visible': <true>, 'count': <1>, 'count-visible': <true> }".

Putting all this together, this will set the a Download Progress indicator for ProtonUp-Qt on a Task Manager, if it is running, to 60% completion.

Emitting DBus Messages with PySide6 and QtDbus

Qt has some built-in classes for working with DBus. Since this is all still very low-level they are mostly wrappers around various DBus actions, and still very close to their C++ counterparts.

I didn't have the benefit of all of the above knowledge or how to implement in PySide6 at first, so a lot of this I figured out by hacking away in Python. The main way we can emit this signal in PySide6 and QtDBus is as follows, using QtDBus.QDBusConnection and QtDBus.QDBusMessage.

• Get a handle on the session bus with QDBusConnection.sessionBus() (i.e. session_bus), as we will be sending the message over this bus.
• Create a QDBusMessage object (i.e. messageObject), which we can do with QDBusMessage.createSignal(). It takes three arguments
  • path - This would be object-path in the above example, this would be /net/davidotek/pupgui2/Update
  • interface - This is where we deviate. From the above example, this would be com.canonical.LauncherEntry. Note how we don't include Update here, we just specify the Interface.
  • name - This is where we include the Update part.
    • When using gdbus we would send this all as --signal com.canonical.LauncherEntry.Update, but here we have to split this up.
    • This tripped me up and I spent a while trying to figure out why I couldn't get the code to work, I thought the name argument could be a name that we came up with and that the interface was the same as --signal in the gdbus command. But I eventually figured it out by re-reading the OP for https://github.com/ValveSoftware/steam-for-linux/issues/4777 and referencing https://askubuntu.com/questions/65054/unity-launcher-api-for-c/65199
• Add the arguments data (the application:// URI and dictionary with the download progress information) to the message with messageObject.setArguments(). The above only created the message with information on where to send it but not the data; like writing the recipient information on a letter but not putting anything inside!
  • setArguments expects a list, and this list can contain anything. This is is because any signal could expect any data in any form.
  • For us, it just so happens that we need to send two types of information: a string for the application identifier, and a dictionary with data to Update on the LauncherEntry. - Again, this is specific to this signal, and other signals may take different pieces of information.
  • So we would want to send the following data to mirror the earlier gdbus example: QDBusMessage.setArguments([ 'application://net.davidotek.pupgui2.desktop', { 'progress': 0.6, 'progress-visible': True } ]).
    • These arguments are equivalent to the "application://net.davidotek.pupgui2.desktop" "{'progress': <0.6>, 'progress-visible': <true> }" lines in the gdbus command. Note how in Python we don't have to use any of the weird syntax around the arguments, we can just pass them as-is and Qt takes care of the rest.
• Now we can send the message along the session bus with session_bus.send(messageObject). That is exactly how we mirror the above gdbus command
  • It is also a good idea to check if the bus is still alive/was ever valid, with session_bus.isConnected(), before sending any messages over that bus.

Implementing in ProtonUp-Qt

So the first part covers how we would generally send this message along dbus from the command-line. Then we get more specific in how to do this in PySide6 with QtDBus and its quirks compared to the command.

So for ProtonUp-Qt, I implemented the download progress functionality by by doing this:

• The above PySide6 explanation breaks down how to send the message we want over DBus. So I made a util function in a new util file called dbusutil.py that abstracts this away. Since there's a lot of boilerplate in creating and sending messages, and basic messages like this have the same format, we have a util function called create_and_send_dbus_message that just takes some basic information and gets the bus, creates the message, adds the arguments, and sends the message for us. It does what was described in the above PySide6 bullet points in one helper function.
  • We create some constants to help us with this, mainly DBUS_DOWNLOAD_OBJECT_BASEPATH which is /net/davidotek/pupgui2. This means any DBus signals we send will be prefixed with this, making them more identifiable. When sending a message, we give an object string parameter that gets appended onto this. For example, Update, which would make our path /net/davidotek/pupgui2/Update.
  • We also have DBUS_APPLICATION_URI, which is our application://net.davidotek.pupgui2 identifier.
  • This function checks to make sure the bus is connected for us to make sure messages are sent only if a valid bus is connected. It also
• To avoid having to do a lot of duplicated calls to a general create_and_send_dbus_message, I created another util function that wraps around this called dbus_progress_message, which is specifically for sending the progress message over DBus. This function handles creating the arguments for us and making sure the data we send to the Update signal is formatted properly (being a list with a specific string and a dict with specific data). It also handles setting whether the progress-visible and count-visible parameters should True or False, so we don't have to do that ourselves.
  • Instead of having to create and manage all of the other boilerplate needed when creating a message, or doing it all in MainWindow#send_dbus_download_progress (see below), we can put it all in dbus_progress_message. All we should really have to know to send to send the progress signal is that we need to give it progress and count which is the data displayed. Everything else should be taken care of in dbusutil#dbus_progress_message.
• Those two util functions handle crafting and sending the message for us. We actually call them in pupgui2.py though.
• We have a new method in MainWindow called send_dbus_download_progress. This handles massaging the progress and count variables to send to dbusutil#dbus_progress_message by making sure progress is 0-1, and sending count as len(self.active_downloads) as this is a list of active downloads and not an actual count. It then calls dbusutil#dbus_progress_message.
• So now we have a MainWindow method that can send the data to our DBus helper functions. We call our MainWindow method in three main places:
  • MainWindow#set_download_progress_percent - This is where we update the progress bar, so after we do so, we send the data here to DBus.
  • MainWindow#__init__ - We call this on startup of ProtonUp-Qt with a progress of -1 and a count of 0. The reason we do this is because the DBus data is not cleared automatically, we have to "clear" it by setting the progress-visible and count-visible properties to False. In order to make sure we properly do that, we call the DBus function with those values to clean up and hide any previous progress data we sent to DBus. This means if ProtonUp-Qt crashes or otherwise doesn't exit cleanly (Steam Deck dies mid-download, power outage, system crash, etc), then the next time we open ProtonUp-Qt, we will clear the previous download progress data we sent to DBus and so there won't be a progress bar or download count displayed.
    • We set 'progress-visible': progress >= 0 and progress < 1, so if progress is -1, it gets set to False, so the progress bar is hidden.
    • We set 'count-visible': count > 0, so if we send count as 0, this will be False and the count indicator will not be displayed.
  • pupgui2#main - We call this function with a progress of -1 and a count of 0 right before we call sys.exit at the end of the main function for the same reason we call this util function on startup: To make sure we clean up and remove any download progress on application shutdown, as without this, the progress bar and counter would show up with the same data when the program opens again.

Concerns

I made a few decisions when implementing this PR that I wanted to address.

Changes to MainWindow#set_download_progress_percent

I made some changes to set_download_progress_percent to use a full if/elif/else, and I removed the returns. This was because I want the method to always fall through and call dbus_progress_message. If a download is cancelled or fails it means we will properly send the negative progress and updated active downloads to DBus via dbus_progress_message, as otherwise the progress would hang. The alternative would've been putting calls to dbus_progress_message in each block that had a return.

It doesn't seem to have negatively impacted the progress method, it seems to work as it did before, since value doesn't cahnge inside of set_download_progress_percent so it should be safe. But let me know if these were not an elif for a reason, and we can revert that change and put the calls to dbus_progress_message in each of those blocks explicitly.

DBUS_INTERFACES_AND_SIGNALS Constant

I created a new constant called DBUS_INTERFACES_AND_SIGNALS. This is a dictionary of dictionaries. Each nested dictionary contains an interface and a signal key/value pair, and each of these nested dictionaries is identified with a name representing the signal that the interface and signal belong to.

My rationale for this was, in future if we ever decide to send other information over DBus to other signals, instead of hardcoding the names, we should keep track of them as a constant. And when working with them in Qt, the names are broken down into the interface and name, for example com.canonical.LauncherEntry as the interface, and then Update as the signal.

This means in wrapper functions that wrap around sending a DBus message, such as dbus_progress_message, only that wrapper function has to be aware of what the interface and name are.

However, I am not super happy about this approach. I think we should have some constant to track these pairs, but I feel like having to remember them by their higher dictionary key, such as LauncherEntryUpdate, is a bit flimsy.

Maybe this should be a dataclass/enum or something that keeps track of these names, and then we can make the higher key something like DBusSignals.LauncherEntryUpdate? I am not sure, thoughts on the current approach and any improvements are welcome!

Better Naming for dbus_progress_message

Perhaps we should make this a bit more general, like send_task_manager_progress, since it's on the MainWindow? That way as well, if this ever does get implemented at a higher level in Qt, we don't have a reference to DBus where it is unnecessary.

Efficiency

I believe the current approach to sending information to DBus is efficient. I think you have to keep sending messages to a signal that listens for it, despite the name being createSignal which did have me a little concerned that we should only make one message this way and keep re-using it. But that doesn't seem possible, and messages seem to be one-time objects that you send and dispose of.

But, I knew nothing about DBus until yesterday afternoon. I still don't really know very much, just a very basic understanding on how to send this specific kind of message over DBus. So I want to caveat this PR with the fact that, it is very possible that my approach is not correct. I based it off of my own tinkering around and examples I seen online, primarily in C++, a language I am very unfamiliar with.

So if you spot anything in the code that doesn't look right please let me know!

Systems Without DBus / Restricted Permissions

I believe handling the busses with dbus.isConnected() should be enough, but I am concerned in case the calls to QtDBus altogether fail if DBus is missing. I am not sure if this is the case and don't know how I would test for that, so it may be worth double-checking this.

In particular we may want to be careful with the Flatpak permissions, in case we need to explicitly add permissions.

Future Work

Not much to note here other than, if this ever does get added as a Portal and/or if Qt ever adds a higher level wrapping around this for us, we may want to replace this approach with that.

---

That was quite a lot for what should've been a very straightforward feature to implement. I hope my explanation and rationale made some sense.

All feedback on this implementation is appreciated!

Thanks!

[sonic2kk]

Should also mention that I made dbusutil a separate file just because it makes the code a little cleaner, and I think before we mentioned breaking up the util files a little bit, so it takes us the work later since this is a brand new set of utils unrelated to previous ones.

Maybe after we start introducing unit tests, we can look at breaking up the files a little more :-)

[sonic2kk]

From looking at Flatseal, there is a D-Bus session bus permission that we would probably have to add here as --socket=session-bus based on the Flatseal description: https://github.com/flathub/net.davidotek.pupgui2/blob/2042c0940a5ec0488e0e3c4f2361ba3d1bf9d408/net.davidotek.pupgui2.json#L11

[sonic2kk]

I should've mentioned: If you'd like me to test on other DEs let me know, but I think it should work for the major ones that have functionality which listens for this DBus signal.

It's not my intent to merge something that only benefits two specific Task Managers on KDE. It's my preferred DE but not the only one and my hope was to make something to benefit any Task Manager that can display progress like this. It just happens that I use KDE, and that this DBus signal (while originally made for Unity7) seems to be implemented by popular Task Managers, not just KDE.

[DavidoTek]

Thanks, that is a nice addition!

The tl;dr is that we have to get down-and-dirty with DBus as there is no higher-level wrapper for this functionality that I could find with Qt, at least not in Qt6

I feel like for some of the desktop stuff there are good libraries missing. I think it is similar with tray icons. Last time I checked, there actually were libraries available, but I think the implementation was a bit different for KDE and GNOME for example.

This tripped me up and I spent a while trying to figure out why I couldn't get the code to work, I thought the name argument could be a name that we came up with and that the interface was the same as --signal in the gdbus command

I guess that's just the paradigm with the RPC stuff... I remember I had similar issues when I wanted to implement media control using the MPRIS D-Bus interface for something.

Note how in Python we don't have to use any of the weird syntax around the arguments, we can just pass them as-is and Qt takes care of the rest.

Good. I don't like it when you can't just pass a dict but you need to create a string with a "json-like" object first.

new util file called dbusutil.py that abstracts this away To avoid having to do a lot of duplicated calls to a general create_and_send_dbus_message Should also mention that I made dbusutil a separate file just because it makes the code a little cleaner, and I think before we mentioned breaking up the util files a little bit, so it takes us the work later since this is a brand new set of utils unrelated to previous ones.

I guess we can use that if there is anything else in the future. For example, notifications also use D-Bus (org.freedesktop.Notifications). But I guess there are some wrappers for that already.

This means if ProtonUp-Qt crashes or otherwise doesn't exit cleanly (Steam Deck dies mid-download, power outage, system crash, etc), then the next time we open ProtonUp-Qt, we will clear the previous download progress data we sent to DBus and so there won't be a progress bar or download count displayed.

Oh, I didn't knew that it was persistent. That's interesting.

---

I made some changes to set_download_progress_percent to use a full if/elif/else

I think it is fine to handle all cases of value using if/elif/else. Not sure why I did it like this in the first place. I don't think we should include if len(self.pending_downloads): in that though, i.e., leave if value == -2 as is.

Maybe this should be a dataclass/enum or something that keeps track of these names, and then we can make the higher key something like DBusSignals.LauncherEntryUpdate? I am not sure, thoughts on the current approach and any improvements are welcome!

I think the current approach is okay, but something like an Enum. I think there are some overrides required for an Enum to store anything but integers or strings. What we can do is just have multiple variables like DBUS_LAUNCHER_ENTRY_UPDATE = {...} (or maybe a better name idk).

Perhaps we should make this a bit more general, like send_task_manager_progress, since it's on the MainWindow?

That would make sense I guess. Not sure if we are getting something high-level from Qt in the near future though :)

I believe the current approach to sending information to DBus is efficient. I think you have to keep sending messages to a signal that listens for it, despite the name being createSignal which did have me a little concerned that we should only make one message this way and keep re-using it

I can't answer that for sure, but when looking at other examples, some of them are doing it the same way. https://doc.qt.io/qtforpython-6/examples/example_dbus_pingpong.html does it differently, by first creating a QDBusInterface and then calling ping on it. I wonder if ping and Update work the same.

I believe handling the busses with dbus.isConnected() should be enough, but I am concerned in case the calls to QtDBus altogether fail if DBus is missing. I am not sure if this is the case and don't know how I would test for that, so it may be worth double-checking this.

I tested the code inside the Flatpak by running flatpak run --command=sh net.davidotek.pupgui2 without giving access to the session bus. bus.isConnected() returned True and nothing crashed :) I'm not sure what happens if there is no D-Bus at all. I can't imagine there is any desktop environment that works without it. Not sure though.

---

I should've mentioned: If you'd like me to test on other DEs let me know, but I think it should work for the major ones that have functionality which listens for this DBus signal.

I'm planning to test it on GNOME. I think it is a standard interface and should work on most DEs. I'm not sure what happens when sending the message to an interface that doesn't exist. Apparently to the Qt docs, send just returns False then? Returns true if the message was queued successfully, false otherwise.

[sonic2kk]

doc.qt.io/qtforpython-6/examples/example_dbus_pingpong.html does it differentl

I wonder if it does it differently because it's set up to send and receive signals, whereas we're just sending the message along to (presumably a Task Manager) listening for it. I don't know that for sure though.

I think the current approach is okay, but something like an Enum. I think there are some overrides required for an Enum to store anything but integers or strings. What we can do is just have multiple variables like DBUS_LAUNCHER_ENTRY_UPDATE = {...} (or maybe a better name idk).

I did have a quick look about creating an enum, and we can just create enums that store dictionaries in Python 3.10.14 (I set up a virtual environment with Python 3.10 at long last). This is perfectly valid:

from enum import Enum

class DBusSignals:
    LAUNCHER_ENTRY_UPDATE = { 'interface': 'com.canonical.Unity.LauncherEntry', 'signal': 'update' }

# Gives us the dict
print(DBusSignals.LAUNCHER_ENTRY_UPDATE.value)

If you'd like I can refactor the code to use an Enum here. We could put it in dbusutil or datastructures, or somewhere else. I'm not sure if another type of class makes more sense here over an enum.

I tested the code inside the Flatpak by running flatpak run --command=sh net.davidotek.pupgui2 without giving access to the session bus. bus.isConnected() returned True and nothing crashed :)

Oh wow that is surprising. I wonder why that permission exists for the Flatpak, maybe it is used differnetly than I expected. But that's good to know there are no alarming fires on the Flatpak side to resolve :smile: Thanks for checking that.

---

I think it is a standard interface and should work on most DEs. I'm not sure what happens when sending the message to an interface that doesn't exist.

It should return False yes, and at least will do nothing. I think it is very unlikely that this signal will exist and not do what we expect it to (i.e. a DE has implemented it to blow up the user's computer :laughing:) so the only scenario we need to be concerned with is if the signal doesn't exist, but even then it will simply do nothing. I got the signal wrong many times in testing and nothing happened.

I'm not sure what happens if there is no D-Bus at all. I can't imagine there is any desktop environment that works without it. Not sure though.

I am not sure either, but I'm sure if there are any desktops without dbus we will hear soon enough :smile:

[DavidoTek]

I tested different scenarios and everything seems to be working fine. I also ran it from inside the Flatpak, the progress bar worked there too. It's probably a standard interface. Thanks again, I will merge this now.

If you'd like I can refactor the code to use an Enum here. We could put it in dbusutil or datastructures, or somewhere else. I'm not sure if another type of class makes more sense here over an enum.

We can leave it as it is now. The Dict-Enum feels Python-y, but it's a bit too much in my opinion :smile:

I feel like having to remember them by their higher dictionary key, such as LauncherEntryUpdate, is a bit flimsy.

I just tested it, today's smart IDEs take care of it :tada:

[sonic2kk]

Been using my branch and now main since this merge with this feature and I think it adds a nice little bit of UI sparkle :smile: I hope others like it too.

If ever you want me to revisit the way this is implemented (or anything else) please feel free to ping me.

I just tested it, today's smart IDEs take care of it 🎉

Wow, I had no idea this feature was available in VSCode, I was about to ask what it was but I was able to find it. That is pretty cool!