louislam / uptime-kuma

A fancy self-hosted monitoring tool
https://uptime.kuma.pet
MIT License
56.81k stars 5.11k forks source link

Can/should we add `privacy-respecting usage metrics`? #4456

Open CommanderStorm opened 8 months ago

CommanderStorm commented 8 months ago

Dear community,

I would like to discuss if adding privacy-respecting usage metrics is something we want to learn from and to inform our actions.

This discussion is based on two impulses

Core requirements:

Implementing such metrics would have these concrete benefits:

This would also have downsides:

I would especially appreciate feedback from the regular contributors (I apologize for the ping) @louislam, @chakflying, @Zaid-maker, @marco-doerig, @Saibamen, @Computroniks, @MrEddX, @AnnAngela, @cyril59310, @apio-sys

PS: I know that privacy is a charged topic, but please let's keep the discussion civil ^^

AnnAngela commented 8 months ago

I personally fully support your idea, as long as it's effectively anonymized, and look forward to others' opinions.

cyril59310 commented 8 months ago

I think that collecting data anonymously on the usage of Uptime Kuma is a good idea.

MrEddX commented 8 months ago

Yes, the idea is undeniably good and would be of great benefit to the developers of the project. From privacy or ethical point of view, I think that the end user should be given the right to choose whether this feature is active or inactive, although the data is anonymous.

ddevault commented 8 months ago

Hard requirement: any data collection must be opt-in only.

adding a pop-up for our users to decide if they would like to contribute their data to these metrics (=> "Informed consent")

Perfect, but focus also on the "informed" bit: exactly what data is collected, for what purpose, and how is it used?

Only collecting metrics for concrete experiments/questions.

I agree with this requirement, but it's not well supported by your summary.

Implementing such metrics would have these concrete benefits:

For each of these supposed benefits, you need to make a clearer case in order to justify monitoring. "It would be interesting" is generally not good enough. For example:

Do people use the maintenance system? If yes, how many and how much?

Positing a particular kind of metric you want to track like this is a good start, but expand on it:

ddevault commented 8 months ago

Also be aware that adding these features is going to subject you to the GDPR. You will have to comply with it, which means things like having a publicly accessible data protection officer.

CommanderStorm commented 8 months ago

@ddevault I have reworded the two questions you had problems with.

You are correct, I think a public site explaining with the following content will be necessary

As for tooling, a tool like divviup by the ISRG might be a good choice.

adding these features is going to subject you to the GDPR

Actually, the GDPR only covers personally identifying data. Since I do not intend to ever store such data, such a data protection policy is simple. I have written them before and can do that again. I would recommend you to watch the talk by Will linked above. He goes into plenty of details how this can be done in a manner which respects privacy. When talking about "privacy respecting usage metrics", this is not the same as talking about "spyware" as you referred in https://github.com/orgs/meilisearch/discussions/162. When I talk about usage metrics, his is more nuanced and experiment based. See Telemetry Is Not Your Enemy for an article why "Not all data collection is the same, and not all of it is bad"

As for the duration of collection, I would say this depends entirely on how our users' upgrade behaviour (likely different between major - minor - patch updates) works, as I think only via updates new metrics can be introduced or clientside-disabled. If an experiment has ended, no data is collected and after analysis the data is deleted. => start an experiment, collect results, finish the experiment, publish a new version without said experiment

Sh3nron commented 8 months ago

I support it!

rezzorix commented 7 months ago

I am with @ddevault

If this is implemented then at max opt-in only and with very clear defined limited scope.

mh166 commented 7 months ago

I'm also in favor of adding such telemetry as it will be very beneficial as you laid out nicely. Of course, I agree that it should be Opt-In only.

Thoughts on the implementation

When asking for the admin's consent, please let the initial message be short and concise. From personal experience: the more text there is, the more I suspect it to be because of corporate legal reasons. Therefore i suspect nothing good, am too lazy to read on and just decline.

Thoughts on evaluating the results

Please keep in mind that while this data might help you to prioritize bug fixes or enhancements, new features should still be considered with a reasonable high priority: you cannot measure what is not there yet. 😉 To prioritize between several new features, the number of +1s, together with the number of duplicate issues may be a better indicator.

Another thing to remember when looking at the data: an apparently low usage might not necessarily indicate little potential but might as well show opportunities for UX improvements.

An example from personal experience: when I first started using Uptime Kuma, it was not very intuitive for me to find that there is a simple incident system included. I stumbled across it by accident, then did not remember how I found it in the first place and had to figure out again how I got there in the first place.

The reason: you can only add an incident if you created a status page and if you are visiting said page and if you are editing it right now. Should you not meet either one of these conditions, you may never know about it, because not even the documentation mentions this feature.

Therefore, in this example, a change in UX might also increase the usage of the feature and consequently any prioritization related to it.

rezzorix commented 7 months ago

@louislam I am following the uptime-kuma journey since the very beginning and would be very interested to have your take on all this.

Zaid-maker commented 6 months ago

I am 100% in the favor of adding this feature ☺

rezzorix commented 4 months ago

So this topic here has been unpinned from the main page. I guess we are in the clear now, or is someone implementing this quietly?

Lanhild commented 2 months ago

+1 for implementing such metrics. It would be useful to have some data from "extreme" users that push the limits of the app, that'd help for performance updates.