search

ipfs / ipfs-gui

Creating standards and patterns for IPFS that are simple, accessible, reusable, and beautiful
105 stars 17 forks source link

Define basic metrics and patterns for consent & collection. #125

Closed SgtPooki closed 1 year ago

SgtPooki commented 1 year ago

At a bare minimum, we should be able to get the following metrics for our projects for users that haven't opted-in to collecting the full set of metrics supported by Countly

  1. Weekly/daily/monthly active users
  2. web-vitals (where appropriate) - https://www.npmjs.com/package/web-vitals
  3. download counts (where appropriate)
  4. pageviews
  5. session time

Disclaimer

Note that until this, https://github.com/ipfs/ipfs-gui/issues/130, and https://discuss.ipfs.tech/t/ipfs-gui-metrics-changes-in-progress/15695 have been available to the community for feedback for enough time( see https://github.com/ipfs-shipyard/ignite-metrics/issues/2), we will continue with the implementation we currently have in public-gateway-checker:

Current problems with metrics

See https://pl-strflt.notion.site/Telemetry-b005d4f217f44db3986902c67d922cf4

Current planned solution

https://www.notion.so/pl-strflt/Telemetry-b005d4f217f44db3986902c67d922cf4#0833d8c86b4f4ffc99337305159d27b0

Metrics distinctions

minimal metrics will be 'sessions' and 'views' as categorized by countly (i.e. const minimalFeatures = ['sessions', 'views']). See https://github.com/ipfs/ipfs-gui/issues/130 for more information

Minimal Metrics (opt-out)

We will allow the disabling of minimal features once we switch to opt-out metrics. See https://github.com/ipfs-shipyard/ignite-metrics/issues/2

All Metrics (opt-in)

Note that while these are opt-in, when presented with our cookie banner/modal for the first time (see https://github.com/ipfs/public-gateway-checker/issues/340#issuecomment-1352395130), the checkbox will be enabled for "all" metrics. Closing the banner without other actions will update your consent from "minimal" to "all".

Current planned solution

Describe alternatives you've considered

General

  1. Not collecting metrics
    • Not viable due to the need to prioritize efforts to satisfy customer needs.
  2. Opt-in only metrics (i.e. only send metrics when users consent)
    • Having opt-in-only metrics is not giving us the full picture. See Current problems with metrics above.

IPFS-companion

  1. Using only firefox & chrome store metrics
    • This is no longer viable, as we have no way to normalize these metrics with our other projects' metrics.

Child tasks

I'm proposing the following settings for countly metrics for our projects:

  /**
   * @see https://support.count.ly/hc/en-us/articles/360037441932-Web-analytics-JavaScript-#features-for-consent
   */
  const minimalFeatures = ['sessions', 'views']
  const marketingFeatures = ['attribution', 'users', 'location']
  const performanceFeatures = ['events', 'crashes', 'apm']
  const trackingFeatures = ['scrolls', 'clicks', 'forms', 'star-rating', 'feedback']

  Countly.group_features({
    all: [...minimalFeatures, ...marketingFeatures, ...performanceFeatures, ...trackingFeatures],
    minimal: minimalFeatures,
    marketing: marketingFeatures,
    tracking: trackingFeatures,
    performance: performanceFeatures
  })

Note that from the above list, 2(web-vitals) is not provided by default from countly, and without events in the performanceFeatures group from the above code snippet, we cannot submit custom events/metrics/analytics. However, if we enable events for "minimal" metrics, it's possible that at some point in the future, a custom metric could be sent that does not adhere to our groupings and opt-in ethics. Therefore, 2(web-vitals) above will be considered an opt-in metric only to keep our minimal metrics light.

However, we can use countly's trackView event in order to track page views without exposing the full blown "events" functionality to limit user risk.

SgtPooki commented 1 year ago

related: https://github.com/ipfs/ipfs-webui/issues/1831.

whizzzkid commented 1 year ago

@SgtPooki no geo-location data? others sound reasonable.

SgtPooki commented 1 year ago

@SgtPooki no geo-location data? others sound reasonable.

geo-location data would be excellent for users who opt-in, but this data is primarily focused on non-identifiable data from our apps. I.e. the necessary data we collect even when users opt-out

SgtPooki commented 1 year ago

This work should be done similar to https://github.com/ipfs/public-gateway-checker/pull/309

SgtPooki commented 1 year ago

updated description

whizzzkid commented 1 year ago

@SgtPooki will the basic metrics be recorded when the user is offline and published when the user comes back online?

lidel commented 1 year ago

Gathering metrics/telemetry by default is increasing risk and responsibility (to not send too much), but I understand the rationale that enabling it by default will make it easier to prioritize work.

May not be that controversial, but that depends on execution (what is gathered, what is sent, how it is communicated in our privacy policies). Telemetry gathering is enabled by default in user-focused browsers like Brave (see their P3A) and Firefox (telemetry). IPFS GUIs are a type of similar user agent, so a lot of prior art in browser space, and lessons on what makes good vs bad approach.

For example, in user-respecting browsers it is always possible to opt-out of reporting all metrics. Marking a subset of metrics as "necessary" and saying user "cannot opt-out of these metrics" in IPFS GUIs is.. just unnecessary.

We can get useful telemetry without forcing this on users. :point_right: My advice is to always provide ability to opt-out via Settings – saves time, removes risks.

What are arguments against forced telemetry without ability to fully opt-out?

Depending on whom you ask, this will be either a joke (easy to block using third-party tools), or will raise eyebrows and leave a bad taste in the mouth (shady/dark pattern, or just clear disregard of user agency).

To speedrun the conversation, I summon the usual voices:

I'd say I am mostly a Realist these days, but you may hear from others too. :)

cc @Stebalien as he had stronger opinions in the past about this (https://github.com/ipfs/kubo/pull/6025, https://github.com/ipfs/ipfs-webui/issues/980)

SgtPooki commented 1 year ago

@SgtPooki will the basic metrics be recorded when the user is offline and published when the user comes back online?

countly speaks to this at https://support.count.ly/hc/en-us/articles/360037753291-SDK-development-guide#request-queue, but I am not currently sure if their web-sdk does this. See https://github.com/ipfs/ipfs-gui/issues/130#issuecomment-1352348003

SgtPooki commented 1 year ago

Thanks for the great feedback @lidel.

Just a few comments on some of your points:

There is no such thing as "necessary" metrics that are so essential that users should not be able to opt-out of them.

As a product owner, I disagree, but as a user, I somewhat agree. Protocol Labs and the community is working hard to build tools to get us to the web3 mecca, and without a full view of whether our apps are being used or not, we're working in the blind, on features that could easily be considered irrelevant (without the data to support the need) for the userbase. As a product owner, that's unacceptable. As a user of a tool in an ecosystem, I don't want my information to be profited off of, but I don't want the tool/feature I use to be sunset because they don't know how important it is.

Forcing users to always report back some telemetry along with their IP to a centralized HTTP endpoint without ability to turn it off is.. not the best. :trollface:

While there is always an IP address associated with a request to a server (i.e. client to our countly server), we do no tracking/storing of this information.

most users will not care 🤷, and users who care already have uBlock that will block your Countly metric reporting

Hopefully, opt-out metrics will allow us to get a majority of users metrics so we can make the right decisions for us and them.

At the end of the day, opt-out metrics feel like the right next step, whereas "always-on" necessary metrics can be a potential next step.

For now, following the pattern of brave and firefox is a great plan. I think defaulting to opt-out metrics, and allowing users to disable necessary metrics is a decent compromise we can handle via https://github.com/ipfs-shipyard/ignite-metrics/issues/2. I will update this issue description to clarify that we will allow opt-out of necessary metrics but will have them enabled by default.

If we need to pursue this later due to a lack of metrics for making decisions, we can do that.

SgtPooki commented 1 year ago

Note that further discussion has occurred with mocks at https://github.com/ipfs/public-gateway-checker/issues/340#issuecomment-1352448082

BigLep commented 1 year ago

I see there's been some good discussion here. A few high-level things I'll say:

  1. Yes, we always we need to make it self-service in our UI's to opt-out of metrics. The "realist" perspective above is spot on. We want to prevent misinterpretation that IPFS projects are force collecting user data. Even as product owners, let's not worry about users who opt out. If they want out, we let them out.
  2. I don't think "necessary" is the right term. We don't have any "necessary" cookies or metrics as far as I can tell (e.g., we don't have cookies representing my login session for an account). Have we thought about calling this something else? "base" or "core" metrics (although I get that isn't descriptive for users either). Or "usage" metrics? That said...
  3. Is it really important to be having tiers of metrics? It complicates our implementation and our communication. What about instead simplifying the message to metric tracking on or off. We can make clear what things we do not track or store like IP addresses. Basically, "we're tracking usage so we can make our product better. We never track X, Y, Z. If you don't want to contribute data, feel free to opt out. You can also opt out anytime if you change your mind later." If we can't track additional metrics beyond session/views because of legal reasons though then I would even consider just tracking those basic/core metrics to keep the message simple. I know there are regional restrictions to be mindful of, and I think the nudge to follow the patterns of other respectable user agents (e.g., Brave, Firefox) makes sense.
    • My head isn't fully in this, but my general push here is to towards simplicity even if it causes us to miss out on "richer data" in some cases. Feel free to disagree though if you don't think I'm thinking fully enough about this.
  4. Before we do implementation, I would love to make sure we have a clear design around metrics that is clear about:
    • This is what the user will see
    • This is what will be collected
    • This is what won't be collected If this issue is supposed to encompass this (great), but I wasn't sure it was pulling all this together. I want folks like @lidel to be able to look and give feedback early before we do a bunch of implementation work. That said, if headway was going to be made on it in December while others are out, I don't want to block progress for 2 weeks - carry forward.

Side: "sending non-identifiable metrics by default, and allowing the community to confirm this, is a reasonable expectation of our users" is listed as a "current problem" in the description. Is that one of the problems? If so, let's be more specific about what portion(s) of that statement are the problem.

SgtPooki commented 1 year ago
  1. Yes, we always we need to make it self-service in our UI's to opt-out of metrics. The "realist" perspective above is spot on. We want to prevent misinterpretation that IPFS projects are force collecting user data. Even as product owners, let's not worry about users who opt out. If they want out, we let them out.

force collecting user data is bad, but just so we're clear: that's not what the proposal was. If metrics data contains no user-identifiable data, then in my mind it's not user data, it's app data. Maybe some more work can be done in the future to differentiate between user and app data, but since we're all in agreement on the next step, we can defer that discussion.

  1. I don't think "necessary" is the right term. We don't have any "necessary" cookies or metrics as far as I can tell (e.g., we don't have cookies representing my login session for an account). Have we thought about calling this something else? "base" or "core" metrics (although I get that isn't descriptive for users either). Or "usage" metrics? That said...

I agree. I've been using minimal to describe what used to be referred to as necessary. I would be fine with calling them base or core as well.

  1. Is it really important to be having tiers of metrics? It complicates our implementation and our communication. What about instead simplifying the message to metric tracking on or off. We can make clear what things we do not track or store like IP addresses. Basically, "we're tracking usage so we can make our product better. We never track X, Y, Z. If you don't want to contribute data, feel free to opt out. You can also opt out anytime if you change your mind later." If we can't track additional metrics beyond session/views because of legal reasons though then I would even consider just tracking those basic/core metrics to keep the message simple. I know there are regional restrictions to be mindful of, and I think the nudge to follow the patterns of other respectable user agents (e.g., Brave, Firefox) makes sense.

    • My head isn't fully in this, but my general push here is to towards simplicity even if it causes us to miss out on "richer data" in some cases. Feel free to disagree though if you don't think I'm thinking fully enough about this.

The implementation is already done in ipfs-shipyard/ignite-metrics. The only additional work involved with separating the metrics we collect is toggling on/off individual metrics in the settings screens. For the first round of metrics addition, this toggle is intended only to be sessions and views as that covers the majority of our needs. However, events is something we could talk about including in the minimal metrics.

For ipfs-webui & ipfs-desktop, I was planning on leaving the full metrics we collect today. But for our other projects, I'd rather not overload the metrics we're collecting since we already don't use many of the metrics we collect from ipfs-webui and ipfs-desktop. In addition, I think that for the people who care about what we're collecting, opting out of "all" metrics is a much more likely outcome than opting out of a specifically curated and smaller subset of metrics.

FYI, the only thing I'm aware of that we explicitly emit for ipfs-webui and ipfs-desktop that are not currently in the listed minimal metrics is events.

  1. Before we do implementation, I would love to make sure we have a clear design around metrics that is clear about:

    • This is what the user will see
    • This is what will be collected
    • This is what won't be collected If this issue is supposed to encompass this (great), but I wasn't sure it was pulling all this together. I want folks like @lidel to be able to look and give feedback early before we do a bunch of implementation work. That said, if headway was going to be made on it in December while others are out, I don't want to block progress for 2 weeks - carry forward.

This was the plan, and I was trying to speed through things while covering my bases. Since the holiday was unproductive for me, I will be creating a readme in ignite-metrics (see https://github.com/ipfs-shipyard/ignite-metrics/issues/29) that will cover the bullet points you listed.

Side: "sending non-identifiable metrics by default, and allowing the community to confirm this, is a reasonable expectation of our users" is listed as a "current problem" in the description. Is that one of the problems? If so, let's be more specific about what portion(s) of that statement are the problem.

Fixed, and linked to Notion doc.

SgtPooki commented 1 year ago

Commenting here to update based on my convo from legal:

If metrics data contains no user-identifiable data, then in my mind it's not user data, it's app data.

This is the legal perspective as well.

olizilla commented 1 year ago

since we already don't use many of the metrics we collect from ipfs-webui and ipfs-desktop.

If we're still not using them then let's remove it. I took a lot of care in adding those metrics but I regret it. They were not connected to a set of questions that we desperately needed answers to in order to make better tools.

SgtPooki commented 1 year ago

@olizilla the metrics are useful, but honestly we don't have the bandwidth to do analytics on all the data we have. It's useful when we are curious, but we haven't made any business decisions in the last year based on anything other than sessions/views/events.

Also, we have a set of questions we're planning on answering defined at https://github.com/ipfs/ipfs-gui/issues/126#issue-1467051352