Metric collection system

[Popolechien]

Folks at SolarSPELL run a similar project to our hotspot and have implemented a basic usage tracking feature: mdseiler/SolarSPELL.

Could we easily implement it into our own code?

[rgaudin]

Also worth mentioning OLIP plan

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[Popolechien]

Here's a sample output from SolarSPELL usagedata-10-22-2019 VU3.csv

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[rgaudin]

We need specifications listing what you'd like so we can start a discussion on what's possible and how. We all know there are multiple ways around metrics:

• capturing metrics (we don't do it yet)
• reporting metrics to hotspot users. OLIP does it with GoAccess but reported they're not happy with it (we should follow up)
• uploading metrics and/or analysis to a central location for later access/analytics. OLIP does it already with a custom solution.

[letompouce]

System metrics: OLIP uses a custom script to gather some system logs, however I'd like to leverage existing tools such as sosreport. We also have some basic battery monitoring script ; maybe we should also monitor the hotspot connectivity itself, and other things. By the past we built a quick PoC using telegraf but didn't pushed this path very far.

Web logs: OLIP implemented a GoAccess based report, which is basically a web logs stats system ; we added a poor's man app specific filtering but that's far from being "business metrics", for which the instrumentation should occur in OLIP (and/or Kiwix!).

At the end, we rsync all of this for a future analysis - which means such metrics/logs must be stored on the device until an internet connection allows upload.

About what business metrics would be needed, I feel we should rely on our MnE people expectations. Adding this to my todo list.

[rgaudin]

Thank you for this detailed feedback @letompouce. I understand OLIP users have access to those 4 reports on their box with some metrics (looked at the screenshots in the documentation) giving an idea of the general use.

I am curious to know if this data is already leveraging actions both at the deployment level (I doubt it would tbh) and at your central level. What use are you making of those system logs? Has it ever been useful for maintenance for example?

[letompouce]

It happened we were asked to provide for an overview of a device/project usage ; I think that's pretty much it about the web stats.

The system logs are quite useful for troubleshooting when the device is not connected permanently ; it helps to have some insights about the problem so we can act quickly once the device gets connected.

The logs are also useful when we wonder about global usage. For example, the overall time spent online vs offline ; the disk storage usage ; etc

[kelson42]

By reading the comments, I see:

• We have different sources of log: system, app, etc...
• We should be able to read reports about then onprem and in the cloud
• What kind of visualisation or important information the reporting dashboard should show it very unclear so far.

Therefore at this stage, the only thing we are basically sure is that:

• we should gather logs, web logs being probably the most important
• we should find a way to upload them (a job for the hotspot companion IMO)
• we need a basic solution (at least first) to visualize the log onprem and in the cloud

Doing this (see point above) would be IMO the first step(s) to do.

[kelson42]

Based on previous comment, and after discussion with @rgaudin, I continue my design work for a first milestone:

• we would have a stat dasboard for local hotspot admins
• gathering web logs is necessary, at proxy level, and we should define where to save them (on local external USB storage?) and how to manage their lifecycle.
• At that time it almost impossible to have clear requirements from the field about what kind of charts we should provide, therefore we should setup some kind of base requirements ourself. Kind of best-effort and we know this might evolve a lot in the future and be prepared to that.
• I believe what the users want is something pretty high level, to a large extend without straight relationship to the logs.
• Point above put in relationship to the fact that sophisticated log analysers are resource intensive, let me think we need something differrent (than a standard log analyser)
• The best technical approach might be to have something like a log indgester library like https://github.com/waveform-computing/lars and custom code to compute our KPI values. We could then generate more or less static data files in json. Finally have a static dasboard using one of such frameworks https://vuejsexamples.com/15-best-vue-js-admin-dashboard-templates-in-2022/ for the rendering.
• Described plan would be modular and really light to deploy/run. The code to compute the KPI is the key point in that architecture.

[kelson42]

Here would be a base requirement in term of dashboard (form a user perspective):

• A dashboard with a list of modules in the sidebar.
• Each module has its own sidebar menu entry.
• A module can be either global (no filter, all the stats), or a custom one which is basically computed on the subset of the logs (URL prefix might be a simple filter to consider). We might consider to allow the admin to create each own module (would be as simple as a label and a regex).
• Each module command the display of a list of widgets/charts which basically are always the same (between modules)
• These widgets can apply on different time ranges: last days, last week, last month, last year, since ever. Probably to be decided for the whole dasboard.
• Here is a brief list of charts/widgets we could start with: unique visitors, visit duration, top pages visited, time of visite the day

@Popolechien @rgaudin Could we start with this?

[rgaudin]

• I disagree with the global/filtered feature. There's no clear, immediate benefit from it so it's not for a first version. Expecting users to input regexp (on what field?) seems quite disconnected from who our users are.
• I understand modules are kind-of mini dashboards: a collection values/displays for specific KPIs.
• I think the time ranges should be flexible: you can use such shortcuts or select specific date(s)
• I'm not fond of the visit duration. We'll probably don't get it with a light log ingester as described above and it's usually too approximate to be useful. Users are not technicians and I'm afraid they might trust this untrustworthy data.
• More than anything, I think we need a list of content (packages) accessed with their number of visits and the distribution.

[kelson42]

• I disagree with the global/filtered feature. There's no clear, immediate benefit from it so it's not for a first version. Expecting users to input regexp (on what field?) seems quite disconnected from who our users are.

I guess user wants to have detailed view based on app/content... not only a global view.

The regex system is only how it works. You can decide for the user what are the modules, or ask him to just put a URL prefix, or ask him a regex. The point is: it's a URL filter. That way you can make a module pro app (everything served via kiwix-serve instance) or you can make a module based on a special content (requests made to a specific zim file).

• I understand modules are kind-of mini dashboards: a collection values/displays for specific KPIs.

It is exactly the same dashboard like the global one, just not based on the same data: made on a subset of data.

• I think the time ranges should be flexible: you can use such shortcuts or select specific date(s)

It's impossible to make static dataset if you have flexible data range. Not worth it IMO.

• I'm not fond of the visit duration. We'll probably don't get it with a light log ingester as described above and it's usually too approximate to be useful. Users are not technicians and I'm afraid they might trust this untrustworthy data.

I have no strong opinion on this. It is IMO a complex discussion, lets discuss this separatly.

• More than anything, I think we need a list of content (packages) accessed with their number of visits and the distribution.

Good point! Might be a "global" only widget/chart (comparison between modules).

What is "the distribution"? Distribution over what exactly?

[rgaudin]

Ok I understand what you mean now. So with what you described we'd have only one module that is loaded with either the whole dataset (global) or a subset based on a prefix 👍

I believe flexible dates are very important. Periods mean having wanted data and unwanted (because of lack of precision) this noise and this uncertainty about the output. Our biggest users work around activities that are date bound and being able to assess them will be really useful. I don't think it will use much resources to do this dynamically but we should indeed choose one of those strategies before starting implementation.

Regarding distribution I meant over other packages/url. Numbers are useful but a pie chart showing that package X getting 30% of all traffic is direct, actionable information.

[kelson42]

I don't think it will use much resources to do this dynamically but we should indeed choose one of those strategies before starting implementation.

We will have to discuss a bit that point then :)

One possible compromise approach might be to declare the event a-priori, and not a posteriori.

Regarding distribution I meant over other packages/url. Numbers are useful but a pie chart showing that package X getting 30% of all traffic is direct, actionable information.

Definitly agree!

[Popolechien]

Numbers are useful but a pie chart showing that package X getting 30% of all traffic is direct, actionable information.

I agree with this. At this (early) stage I would favour a dashboard that may be less customizable but that strongly emphasizes a visual display of whatever data is collected (pie charts, etc.).

[rgaudin]

Similar project https://datapost.site (using https://www.chartjs.org/ for nice charts)

[Popolechien]

re: datapost, I've reached out the folks at World Possible and here's what they had to say:

We haven't open-sourced the DataPost code, we're still sorting through our roadmap for the product really. We do plan to release an installer for other devices at some point in the future, there are just lots of considerations around how we would run DataPost on devices that don't have an internal battery or real-time clock we haven't started to tackle yet.

[kelson42]

Nice... we will have to partly redo what they are doing.... looks like the FOSS spirit goes one way with them

[Popolechien]

Comment from user (Literate Earth, running schools in East Africa):

primarily we wanted to be able to track engagement with Wikipedia for Schools. I think this would be number of searches and number of clicks on hyperlinks.

[benoit74]

I've began to assemble some implementation details here: https://github.com/offspot/container-images/tree/metrics/metrics

[rgaudin]

Superseded by https://github.com/offspot/metrics/issues/1. Closing this so we have a single discussion channel