Enhance User Statistics in WebUI

multiduplikator commented 3 years ago

qBittorrent version and Operating System

4.3.1+/any OS

If on linux, libtorrent-rasterbar and Qt version

any

What is the problem

Traffic related User Statistics available in WebUI are very limited, i.e. only all-time up/down and share ratio

What is the expected behavior

It would be great to have at least the same capabilites as demonstrated in the traffic plugin of rutorrent, where up and download stats can bee segregated by up/down/tracker and aggregated on day/month/year level. See: traffic plugin

Steps to reproduce

Open WebUI > View > Statistics

Extra info(if any)

n/a

multiduplikator commented 3 years ago

Is there a chance I can place a bounty on this soon?

denim2x commented 3 years ago

I'm interested in this

multiduplikator commented 3 years ago

You mean solving for the bounty? Great.

Maybe this can be solved in 2 steps:

Make available the required dataset in long format data frame, maybe in csv format to display/download. By doing so we could play with the data frame and see what can be done with it, and maybe spot missing data. Also, we could generate some sample charts and tables and see what would be a reasonable default.
Add default charts/tables

That would satisfy the requirement, I guess. As an improvement item, I would add some selectors, where one can select certain dimensions and ranges of variables.

Since we have data in long format available already in step 1, we open the option of experimentation early and all kind of analytics could be done externally...solid prep for step 2.

What do you think?

oogetyboogety commented 3 years ago

@multiduplikator The libtorrent statistics (counters and gauges) might not have the granularity you are looking for when it comes to torrents and trackers. I know it sends these statistics to other trackers internally, but I'm not sure this gets exposed to the qBittorrent session and then to the WebUI. As such, it might be a better architecture to log the torrents' state at the server update intervals into the dataframe that you are seeking, and establish this in a javascript view/dataframe that can sit alongside the current TorrentsTable. You might query it using something like [4] If you were looking to directly write to the filesystem from the regular qt UI, I think that certain internal APIs would have to be updated to specify a path similar to the current default logging one, although [1] is probably the state of the art approach to tracking metrics in a cloud native way. Per torrent metrics might add some usability risks when it comes to bloat. [2, 3]

[1] https://github.com/esanchezm/prometheus-qbittorrent-exporter [2] https://github.com/tobbez/deluge_exporter [3] https://github.com/tobbez/deluge_exporter#per-torrent-metrics [4] https://observablehq.com/@tmcw/using-observable-with-a-local-server If you would like, I could start work on this issue using the implementation I mentioned (Alternative Web UI fork)

multiduplikator commented 3 years ago

@oogetyboogety Sounds like a good start to me. Eventually, it would be great if that javascript based "page" would show a default graph, besides the raw data frame. Similar to what rutorrent plugin is displaying.

And I fully support the idea not to bloat this beyond usability.

I would be happy if you got started on this. We can discuss options along the way...

oogetyboogety commented 3 years ago

@multiduplikator Sorry about the lag on this one. Had some fires come up at work. Let's iterate and try to wrap this up this weekend/week if you're free. I've had some experience with the PerspectiveJS [1] library in terms of usable analytics applications based on the real time data that fits this torrents metrics statistics use case. The library is split into two parts. The core is a WebAssembly component (WebAssembly, being a newer technology, might allow for large dataframes to take up smaller portions of memory in a more performant native manner than using the browser's Javascript libraries in its VM). This core WebAssembly component contains an internal representation of a table that is logged to in the fork of the default WebUI [2] that I demo [3], and accumulates all the torrent status JSON events over time. The library also contains a component called perspective-viewer.It supports the grid, as well as the graph, scenarios you described in a perspective viewer, however there are blockers I reached involving the mootools-js library that the default WebUI uses. It interferes with the perspective-viewer webcomponent as it uses a custom HTMLElemnt that produces its own DOMEvents that mootools doesn't know how to process. So in essence the performant parts are there in the fork [2] but the attractive elements are not.

At this point, its important to discuss some options along the way and validate the ones I have chosen so far (perspcective JS, WebAssembly, etc.) in the fork described.

Today, I am looking at either making [2] work or just forking the react native or Vue WebAPI clients [4] so that a user can open a different URL to use things such as the perspective viewer in a similar fashion as the attractive alternative webUIs VueTorrent[5] or aminpak/qbittorrent-web-ui[6]. This increases usability by allowing the browser tab to crash without closing the entire WebUI for the user.

@multiduplikator Let me know what you think about the preliminary options Ive chosen and I'll update once I've completed the demo repository. [1] https://github.com/finos/perspective [2] https://github.com/oogetyboogety/www_addDetailedStats [3] https://github.com/oogetyboogety/www_addDetailedStats/blob/master/private/scripts/client.js#L655-L657 [4] https://github.com/qbittorrent/qBittorrent/wiki/List-of-unofficial-WebAPI-clients [5] https://github.com/WDaan/VueTorrent [6] https://github.com/qbittorrent/qBittorrent/issues/14098

multiduplikator commented 3 years ago

@oogetyboogety Darn, I am extinguishing fires as I write...bit of a mashed brain right now.

Nevertheless, I must say that I find perspective [1] to fit the bill quite nicely. It would be great if [2] can be made to work.

How would that fall back scenario [4] pan out then? Would a user have to install an alternative webui all together, or would you only reuse some concepts to provision the "different" URL, but no additional webui would have to be installed?

Shall we hop on IRC?

oogetyboogety commented 3 years ago

@multiduplikator I see a qbittorrent channel on Freenode, but I can wait for your brain to restore some plasticity, because I know exactly how that feels.

However I do think you're following along with what I'm saying.

The first thing we should do to make [2] passable is add a user preference that enables the extra weight it adds with Perspective and the extra tab that it adds to the UI. My fear is there would be other concerns with merging [2] held by the authors. We would have to discuss with them about the design as well.

Let me answer your question on how fall back scenario [4] would pan out. I should illustrate the trade off here, and I think the user aminpaks does an excellent job of this in his repo here [7]. You could use a solution like the development server in that repo with CORS proxy [8] (perhaps make it installable with [12]) and if the tab or node.js client crashes, your qbittorrent Web UI state remains unchanged and you can still access the qbittorrent Web API to sync a full update when the tab restarts at port 8080. The decision to make here is:

whether you want the user to installer another parallel process besides the WebUI [8, maybe 12] or
whether you want to provide the option to the user to install this WebUI instead [9] which also incorporates the statistics UI a user would like to see.

I wanted to spec out the graphs and layout of the grid with you, and in regards to this I was thinking it would be easier to integrate [11]. What do you think about [11]? [7] https://github.com/aminpaks/qbittorrent-web-ui [8] https://github.com/aminpaks/qbittorrent-web-ui#start-the-local-web-server [9] https://github.com/aminpaks/qbittorrent-web-ui/wiki/How-to-use-this-custom-WebUI [10] https://github.com/rmartin16/qbittorrent-api [11] https://vega.github.io/vega/examples/ [12] https://github.com/Metnew/webpack-pkg-plugin

oogetyboogety commented 3 years ago

@multiduplikator current progress:

https://github.com/oogetyboogety/qbittorrent-web-ui

multiduplikator commented 3 years ago

@oogetyboogety Wooohoo! This looks like we are getting real nice data to play with!

PS: In case the bounty deadline expires, I will renew, of course...

oogetyboogety commented 3 years ago

@multiduplikator Would you be able to follow the instructions here: https://github.com/oogetyboogety/qbittorrent-web-ui#development or here: https://github.com/oogetyboogety/qbittorrent-web-ui/releases/tag/0.4.01 to run through an example this here:

https://user-images.githubusercontent.com/3319254/111105276-3df35a80-8518-11eb-8dce-3f78b172250f.mp4

multiduplikator commented 3 years ago

@oogetyboogety Sure, followed the first link of instructions - mostly :)

I edited .env directly instead of creating .env.local - should be ok.

When trying to start, I ran into all kinds of errors, but the following resolved all of them:

npm install string-replace-loader
npm install style-loader
npm install css-loader
npm install @finos/perspective
npm install @finos/perspective-viewer
npm install @finos/perspective-webpack-plugin
npm install @finos/perspective-viewer-datagrid
npm install @finos/perspective-viewer-d3fc

Then I ran into ECONNRESET error on the proxy. This was due to the fact that I had forced qbit to https. So I enabled http again and restarted qbit.

I get on the watch log:

Open the host in browser: http://localhost:9000
Proxy Server: <deleted ip>:9000 -> <deleted ip>:9080

ℹ ｢wds｣: Project is running at http://<deleted ip>:9000/
ℹ ｢wds｣: webpack output is served from /
ℹ ｢wds｣: Content not from webpack is served from /<deleted path>/qbittorrent-web-ui/public
{ env: undefined, isProductionEnv: false }
[React Intl] Line 130: Default messages are not extracted from <FormattedPlural>, use <FormattedMessage> instead.
[React Intl] Line 76: Default messages are not extracted from <FormattedPlural>, use <FormattedMessage> instead.
[React Intl] Line 58: Default messages are not extracted from <FormattedPlural>, use <FormattedMessage> instead.
ℹ ｢wdm｣:    1993 modules
ℹ ｢wdm｣: Compiled successfully.
[09:57:44 GMT+0100 (CET)] Found 0 errors. Watching for file changes.

Which looks ok for me.

Now, I can open the new webui and login and go to the GRAPH section without any errors thrown in the proxy log. Basic charting works as well. Lots of data seems to be missing, which should be there (e.g. category, downspeed, etc.). Also added bigbuckbunny torrent to test ... same thing with the datagrid.

To get the variables that you have shown in your demo datagrid, I guess I have to compile 4.4.0alpha1 ... which would be a bit more challenging for me, but happy to spin up an LCX container and get that done as well, If you need me to.

multiduplikator commented 3 years ago

@oogetyboogety Should I try and build 4.4.0alpha1?

oogetyboogety commented 3 years ago

@multiduplikator definitely should see something similar to variables I had in the datagrid, but probably only after you add bigbuckbunny, as I had it polling for changes and only adding changes to the dataframe. We can add some aggregate statistics which would show for preexisting torrents, but the sync API (definitely the right API for stats, as you can see in [1]) won't get the same granularity on the preexisting torrents (full update vs. partial update in [2]) . I wanted to take care of the torrents added during the session first, because these would have detailed statistics from the sync API we used.

I'm very interested to se if the API has changed from the current stable release to 4.4.0alpha. I saw in some other issue that major changes were made in version 4.2, but nothing after that [3]. There may be some other root cause at play here, so instead of increasing effort on your side to compile the alpha, let me switch to a stable release and test the new WebUI on that version instead. Should be pretty quick for me.

[1] https://github.com/qbittorrent/qBittorrent/issues/12201 [2] https://github.com/qbittorrent/qBittorrent/wiki/WebUI-API-(qBittorrent-4.1)#get-main-data [3] https://github.com/qbittorrent/qBittorrent/issues/10999

multiduplikator commented 3 years ago

@oogetyboogety I will spin up a vanilla qbit instance and try to run the same scenario again - tomorrow I should have news.

oogetyboogety commented 3 years ago

@multiduplikator hmmm when i ran it, it seemed okay,

https://github.com/oogetyboogety/qbittorrent-web-ui/blob/development/perspective.csv

although admittedly all the properties don't fill in every row, the torrent hashes are filled in so that when we run aggregate queries by torrent hash it should return appropriate results

multiduplikator commented 3 years ago

@oogetyboogety Installed latest ubuntu LTS in LXC container, used "official" qbit package repo. Left all at vanilla when setting up the solution as per instructions. I do have proper results now:

oogetyboogety commented 3 years ago

@multiduplikator Awesome! Now that you can see all the metrics, play with the the dataframe, and export the CSV (right click the top left three dots), Is there a default set of graphs and queries you would like to settle on?

we could have an option in the settings of the default UI which would load the perspective library as a performant, stable way to aggregate all the responses the web ui receives from the server over time. This could enable certain graphs that you would recommend in a separate tab in the bottom, I suppose a bit like the speed graph in the QT ui, for example. But i'm just shooting out ideas, it's up to you!

multiduplikator commented 3 years ago

@oogetyboogety For the default set of graphs, I think we could go with the one from the rutorrent traffic plugin for starters.

What I don't understand yet with the solution we have right now, how do we store historic data in an efficient way? Would that be the job of the perspective library or the qbit native somehow. Right now, as I understand, we can only work with data that becomes available as we start the qbit session or even perspective library...

oogetyboogety commented 3 years ago

@oogetyboogety For the default set of graphs, I think we could go with the one from the rutorrent traffic plugin for starters.

What I don't understand yet with the solution we have right now, how do we store historic data in an efficient way? Would that be the job of the perspective library or the qbit native somehow. Right now, as I understand, we can only work with data that becomes available as we start the qbit session or even perspective library...

@multiduplikator Yeah I didn't really think of statistics past the session. You could:

persist the entire dataframe just like the exported CSV above to LocalStorage
set up state in the component using counters and accumulators to persist somewhere, such as LocalStorage.
- The default graph to start with would mean we need to efficiently track average downloaded amount, for example, grouped by tracker and year. I'm not sure what granularity the drop down in the top right goes to but for the sake of a demo I'll just say Hourly. Which statistics would you like to persist, grouped by which categories/levels of granularity, and at what rate?

oogetyboogety commented 3 years ago

@multiduplikator I was also looking at the functionality with Prometheus I mentioned in the above comments. https://github.com/esanchezm/prometheus-qbittorrent-exporter I wonder if packaging Prometheus would be heavy or overkill for scraping this qbittorrent API endpoint when it's not guaranteed to run in in the cloud or in a cluster. For now i'll just aggregate each of the statistics and make it selectable alongside downloaded amount. Hoping to get this solution wrapped up nicely this weekend!

For a charts library, I was probably just leaning towards Vega, which I tend to already use a lot. I'll just start with this.

multiduplikator commented 3 years ago

@oogetyboogety I think it would be enough to persist the amount up/downloaded per tracker per hour - at least for starters.

Granularity As default, I would say that it can be the same as with the default graph from the rutorrent plugin:

You can do "All trackers" or each tracker that you have communicated with.

The time frames are:

Per Year, here the X-Axis granularity then is set to months of the year
Per Month, where the X-Axis granularity is set to days of the moths
Per Day, where the X-Axis granularity is set to hours of the day.

That graph has a somewhat peculiar way of displaying, in that is "rolling forward" over 12 months. Now that we are in March, it would display January, February and March from 2021. And then it would also display Januray-December from 2020. Its somewhat interesting, but I don't think this to be required.

That "rolling forward" feature leads to the two UL and two DL selectors, where you can toggle display of that prior year data.

oogetyboogety commented 3 years ago

@multiduplikator I almost have that entire functionality finished. I'll be updating the WebUI fork https://github.com/oogetyboogety/www_addDetailedStats

oogetyboogety commented 3 years ago

@multiduplikator Wrapping this up shortly, here is what it looks like now, i'm adding the persistence to a file, also adding the time scale form field, add the scale for bytes -> terabytes and i'll style it and then i'll update:

oogetyboogety commented 3 years ago

@multiduplikator Looks like to filter on time scale we have to define some additional behavior. Let's say the filter categorizes hourly and a torrent takes two hours to download.

If the plugin was enabled during the download, it can differentiate how much was downloaded during each hour
If the plugin was not enabled during the download, but the torrent is still in the torrents list, it can categorize the download's size under a timestamp corresponding to either a
- completed_on date or an
- added_on date
  
  and filter the time period on either of those dates.

I've noticed the rutorrent traffic plugin has to be enabled for it to report statistics that it logs.

Also, it appears the qBittorrent API doesn't have amount downloaded to a tracker granularity. It can't tell you which peer sent x amount of data from which tracker's peers list for a torrent, so you have to categorize all downloaded data on a torrent to one tracker. This can be random at times (https://github.com/qbittorrent/qBittorrent/wiki/WebUI-API-(qBittorrent-4.1)#get-torrent-list reports The first tracker with working status. Returns empty string if no tracker is working.) If the download also was enabled for DHT, for example, or a torrent lists multiple working trackers, this might lead to misleading statistics. Just something to watch out for. I don't think there's a simple way around this besides tracking libtorrent counters and gauges in a new way.

Let me know what you think the desired behavior should be. Thanks!

multiduplikator commented 3 years ago

@oogetyboogety Hmm, there we have it, the devil in the details :)

As for the stats per tracker. My thoughts: In standard qbit webui, we get the torrents by tracker in that left side pane. Somehow it decides how to attribute each torrent to one (or maybe more than one) entry in that list. Even if it is not 100% precise, we should aggregate the stats the same way. E.g. if you select a given tracker in that left side pane, you get a list of torrents attributed to that tracker. Hence you should be able to get the aggregated stats for the same. At least we would be consistent this way. Could this be achieved in the approach you have taken so far?

For the time stamp, I somehow lean towards the completed_on date - no particular logic, just a hunch. Open to better arguments :)

oogetyboogety commented 3 years ago

@multiduplikator Yeah that can be achieved! An additional table has to be joined for non aggregate statistics which would record all the trackers for a given torrent. I checked and it looks like the trackers list on the left for the filter of the torrents list actually keeps a separate list of all the trackers for each torrent in the torrents list (it's more than one tracker per torrent on the left), so we just have to persist that data as well.

lemme use the completed_on date in the filter. Will update once I'm finished. Expecting to finish tonight or tomorrow night.

multiduplikator commented 3 years ago

@oogetyboogety How are things going? Shall we finalize this, so the bounty can be awarded?

IcedQuinn commented 2 years ago

@oogetyboogety How are things going? Shall we finalize this, so the bounty can be awarded?

Seems the issue has gone to sleep?

oogetyboogety commented 2 years ago

@multiduplikator Should be able to take a look and finalize this!

multiduplikator commented 2 years ago

@oogetyboogety Yes is seems so. The bounty dollars can be claimed for the work done up to this point, or you can reassign them. I am not sure if and how to manage those now. Its so long back ... but there is a significant amount for you to redistribute.

How shall we proceed?

qbittorrent / qBittorrent