More accurately record time spent consuming video media

[douglasg14b]

The Problem

One of the larger time-sinks today is video. Be that through streaming services like Netflix, Amazon Prime Video, or HBO. Media sites like YouTube, Twitch, or Vimeo. Or from downloaded or streamed media on players like VLC or MPV.

Activitywatch, unfortunately, fails to effectively record the time spent on these activities as it relies on mouse and keyboard input to determine activity. This means when watching a video, Activitywatch will mark the time as afk after a short while. Even though the user is present, and spending time on an activity at their device.

This was brought up in https://github.com/ActivityWatch/activitywatch/issues/186 which was marked as wontfix. I believe that this something that CAN be solved, and should be seriously considered given the amount of time that can be spent consuming media.

Afk time can be disabled, but this then pollutes the data. Users may go afk for a variety of times in a variety of applications or websites throughout the day, which could pollute the afk time to the point of video-specific time may no longer be useful.

Possible solutions

Note: Not all problems/disadvantages/pitfalls are meant to be solvable. I am including them for devils advocates sake and to foster a more robust discussion.

Application/Site Tagging

Compile a list of common media applications and websites. When the use goes afk on this site or application, mark the time as non-afk. This isn't technically tagging, but it could be setup to work in a tag-like way, which would make this into a very extendable solution for more than just videos.

Advantages

• Easy to maintain
  • Add sites/applications as you see fit, or as the community points them out
• Simple

Disadvantages/Pitfalls

• Initial list will be a chore to create
  • Can always ask the community for assistance. You create the functionality, and get the community to fill it in with you curating.
  • Is not quite perfect.
  • What if the user is legitimately afk on a youtube page?
  • What if the user is afk on the homepage of a media site, and not on an actual video?
  • What if the user is afk on a video, and it's paused?

Enhancements

These are here to try and solve for some of the problems presented under disadvantages.

• User-defined or user changeable lists and filters
  • The repo still maintains a list, but users have the ability to add items to theirs without waiting for a pull request to merge and a new release.
  • There are a variety of flavorful solutions to this that I will not go into
• Combine with monitoring hardware to detect if a video might be paused

Monitoring Hardware

Monitor audio output to see if a video is playing.

Advantages

• Simple. May not be easy to implement, but is a simple solution to the problem

Disadvantages/Pitfalls

• What if the user is listening to music, like spotify, and is afk on a media player?
• What if the user is playing music from a media site like YouTube?
• Difficult to implement?
• Monitoring multiple audio devices necessary?

Enhancements

• Combine with site/application tagging

General Enhancements

These are enhancements that could apply to any solution, to increase accuracy and to enable the user to correct mismatches and errors.

User-Defined Lists & Filters

Let the user create/modify the list of sites/applications, and/or the patterns used to match them.

• Takes burden off of developer to maintain accurate lists and patterns to meet everyone's specific needs
  • I would still expect than an "official" or pre-defined list of items and patterns would be bundled or downloadable
• Some passionate users can open issues to merge items and patterns back into the repo with the dev curating them, or just to share them.

Tagging and Pattern Matching

This goes above and beyond, but would really turn this into a much more powerful tool.

Instead of just solving the video problem. Create an extendable solution that encompass the general problem category that the video problem is part of. This would be in the form of tagging, being able to automatically tag domains & applications with predefined or user defined tags. This can be facilitated with pattern matching lists, and depending on the data's schema/format could be applied to already existing data greatly enhancing it's utility.

As an example, time in VLC, YouTube, or Netflix could be tagged as video, which gives users the power to filter this time separately, combine it in reports, or to more easily correct collection errors.

This of course could be setup to be user-manageable, with points listed in the previous section.

Conclusion

I believe the ability to capture time spent consuming video-based media will have an impact on the future usability of this project as these sorts of services continue to expand and bring in more and more people. Solving this problem can not only provide a solution to this problem, but could also greatly enhance the utility and power of this application.

What are your thoughts? (please to not be automarking this as closed, this took some time and effort to create).

[ErikBjare]

I've been thinking of adding a "always count as active" regex configuration rule (possibly with a max-duration, such that actually long AFK periods don't get included by accident).

This would be easy to implement and work as a decent workaround, I think.

Would also help in some situations I've noticed where gamepads aren't detected.

Edit: basically what was suggested by @placeybordeaux https://github.com/ActivityWatch/activitywatch/issues/261#issuecomment-687868006

[marco-coraggio]

That would be great, I would be very grateful if you could do that!! Thank you in any case.

[8Dion8]

I've been thinking of adding a "always count as active" regex configuration rule (possibly with a max-duration, such that actually long AFK periods don't get included by accident).

I feel like that would be a great temporary solution for now. Tried messing around with manual SQL queries to edit existing afk data, and got nowhere. If there was a switch for certain applications, not even in the trackers but in the UI (which would probably be evevn easier to implement and doesn't actually modify data for now), then it would be great

[Aziks0]

@8Dion8 It has been implemented, see ActivityWatch/aw-webui#375.

Just need to wait for the next release now 👀.

[8Dion8]

Awesome! Can't wait to see how much time I actually wasted on binging shows😂

[Valentin-N]

I'm testing the new feature in https://github.com/ActivityWatch/aw-webui/pull/375 which landed in v0.12.2b1 and I am seeing failures that cause the whole Activity page to stop loading.

2023-01-18 22:15:41 [ERROR]: Exception on /api/0/query/ [POST]  (aw-server:1449)
Traceback (most recent call last):
  File "flask/app.py", line 1517, in full_dispatch_request
  File "flask/app.py", line 1503, in dispatch_request
  File "aw_server/rest.py", line 46, in decorator
  File "flask_restx/api.py", line 405, in wrapper
  File "flask/views.py", line 84, in view
  File "flask_restx/resource.py", line 46, in dispatch_request
  File "aw_server/rest.py", line 323, in post
  File "aw_server/api.py", line 310, in query2
  File "aw_query/query2.py", line 417, in query
  File "aw_query/query2.py", line 391, in interpret
  File "aw_query/query2.py", line 140, in interpret
  File "aw_query/functions.py", line 88, in g
  File "aw_query/functions.py", line 117, in g
  File "aw_query/functions.py", line 198, in q2_filter_keyvals_regex
  File "aw_transform/filter_keyvals.py", line 28, in filter_keyvals_regex
  File "aw_transform/filter_keyvals.py", line 28, in <listcomp>
  File "aw_transform/filter_keyvals.py", line 26, in predicate
KeyError: 'title'

[cjc7373]

Just a few notes about mpv under Linux wayland environment. Its manual says:

Disabling Screensaver By default, mpv tries to disable the OS screensaver during playback (only if a VO using the OS GUI API is active). --stop-screensaver=no disables this.

A common problem is that Linux desktop environments ignore the standard screensaver APIs on which mpv relies. In particular, mpv uses the Screen Saver extension (XSS) on X11, and the idle-inhibit protocol on Wayland.

Before mpv 0.33.0, the X11 backend ran xdg-screensaver reset in 10 second intervals when not paused in order to support screensaver inhibition in these environments. This functionality was removed in 0.33.0, but it is possible to call the xdg-screensaver command line program from a user script instead.

So maybe we can utilize this "idle-inhibit protocol". And this protocol is tied to the idle notify protocol, which says:

If an idle inhibitor is active (e.g. another client has created a zwp_idle_inhibitor_v1 on a visible surface), the compositor must not make the notification object idle.

So if we implement idle notify in the afk watcher, it just works! Nothing need to be handled.