search

ipfs / ipfs-gui

Creating standards and patterns for IPFS that are simple, accessible, reusable, and beautiful
105 stars 17 forks source link

Feature: Ensure we have opt-out metrics for all of our projects #105

Closed SgtPooki closed 1 year ago

SgtPooki commented 2 years ago

eta: 2023-01-31

Follow https://github.com/ipfs/ipfs-gui/issues/125 for a list of the metrics that should be implemented within each project.

### Tasks
- [x] #129
- [x] https://github.com/ipfs-shipyard/ignite-metrics/issues/32
    - [x] https://github.com/ipfs/kubo/issues/9502 - separate webui metrics for kubo
- [x] https://github.com/ipfs/ipfs-companion/issues/1115
- [x] https://github.com/ipfs/public-gateway-checker/issues/306
- [x] https://github.com/pln-planning-tools/StarMaps/issues/173
- [x] https://github.com/ipld/explore.ipld.io/issues/106
- [x] https://github.com/ipfs-shipyard/ipfs-dag-builder-vis/issues/38
- [x] https://github.com/laurentsenta/pl-diagnose/issues/10
- [x] https://github.com/ipfs-shipyard/pinning-service-compliance/issues/272
- [x] https://github.com/multiformats/cid-utils-website/issues/56
- [x] https://github.com/ipfs-shipyard/ipfs-check/issues/28
- [x] https://github.com/ipfs/ipfs-desktop/issues/2370
- [x] https://github.com/ipfs/ipfs-webui/issues/2085
- [x] https://github.com/ipfs/ipfs-webui/issues/2074
- [x] https://github.com/ipfs/public-gateway-checker/issues/340
BigLep commented 1 year ago

Great to see this initiative.

For web-ui, will we able to differentiate between https://webui.ipfs.io/ and the webui that is bundled in Kubo?

SgtPooki commented 1 year ago

Great to see this initiative.

For web-ui, will we able to differentiate between https://webui.ipfs.io/ and the webui that is bundled in Kubo?

I hadn't planned to do that work with this initiative but I'll add an item under webui. See https://github.com/ipfs/ipfs-webui/issues/2074

BigLep commented 1 year ago

@SgtPooki @tinytb : a few thoughts before this rolls out to more places:

  1. I wanted to make sure some of the questions that could come to mind for users were seen: https://github.com/ipfs/public-gateway-checker/pull/309#issuecomment-1338014387
  2. There were some stylistic suggestions in https://github.com/ipfs/public-gateway-checker/pull/309#issuecomment-1339261387 that likely make sense to incorporate for all properties
  3. For companion, capturing clicks into Companion makes sense and is good. The other metric that would be great is the number of ipfs:// URLs that it handles (and ideally breakdown between gateway domain: localhost, ipfs.io, Cloudflare, etc). The point is that someone could not be "engaging" with the plugin directly in a given week but still be an "active user" because they're getting value out of the plugin that week because the extension is resolving ipfs:// URLs.
aschmahmann commented 1 year ago

@BigLep @tinytb @SgtPooki any chance we could get https://check.ipfs.network/ (https://github.com/ipfs-shipyard/ipfs-check) added to the list of things we should have metrics for? Metrics here could be frontend, backend or both but generally serving the same point of showing how these resources are being used.

SgtPooki commented 1 year ago

@BigLep

  1. I wanted to make sure some of the questions that could come to mind for users were seen: https://github.com/ipfs/public-gateway-checker/pull/309#issuecomment-1338014387

Answered in https://github.com/ipfs/public-gateway-checker/pull/309#issuecomment-1352451253 but #125 has the most up to date information. Regarding metrics breakdown. We will also be moving to more of a modal view (see #340). These updates will be in effect before adding to other projects (except companion, but companion will have a breakdown)

  1. There were some stylistic suggestions in https://github.com/ipfs/public-gateway-checker/pull/309#issuecomment-1339261387 that likely make sense to incorporate for all properties

Absolutely.

  1. For companion, capturing clicks into Companion makes sense and is good. The other metric that would be great is the number of ipfs:// URLs that it handles (and ideally breakdown between gateway domain: localhost, ipfs.io, Cloudflare, etc). The point is that someone could not be "engaging" with the plugin directly in a given week but still be an "active user" because they're getting value out of the plugin that week because the extension is resolving ipfs:// URLs.

This is a great callout but may not be in for initial launch. We need to tread carefully with metrics we send about usage not related specifically to the app, and ensure things are in place to prevent tracking pageUrls with(or without) CIDs, so that's my main focus for initial opt-out metrics release with companion.

I'll create a section in the metrics collection notion doc (link will be provided in #125 shortly) to address this.

The most difficult problem is getting opt-out metrics in place and ensuring users are aware of our changes. Once that is done, modifying the type of metrics we send can be done fairly easily; though it will still require updating documentation about metrics sent for that project.

SgtPooki commented 1 year ago

@aschmahmann @tinytb

any chance we could get https://check.ipfs.network/ (https://github.com/ipfs-shipyard/ipfs-check) added to the list of things we should have metrics for? Metrics here could be frontend, backend or both but generally serving the same point of showing how these resources are being used.

Do you have a countly account? We don't technically own ipfs-check (though maybe we should) and so if no one on our team will be viewing the metrics and generating reports it won't benefit us much.

I will add it to the list but my primary focus is getting metrics for the following ignite team projects:

And then ensuring desktop & webui are switched to opt-out.

BigLep commented 1 year ago

@SgtPooki : I'll read/watch https://github.com/ipfs/ipfs-gui/issues/125 as it sounds like that's where more of the action is on specifics.

I do think we should add IPFS Check (and pl-diagnose) to the list and just use Ignite's account. You could say that the "owners" should do it, but they don't really have team ownership. Getting the metrics in place heps give clarity on whether we should invest more in them, and if we invest it likely makes sense for it to happen in Ignite (accompanied with increasedresourcing or adjusting other priorities).

They can be lower on the list, but given diagnostic tools are the kind of things we believe users across implementations find valuable (and that they reduce the burden on core maintainers) taking the 1-2 days to measure usage I think is worth it.

SgtPooki commented 1 year ago

note that telemetry documentation is merged and #125 closed. Please open an issue on ignite-metrics if there or questions/comments!

SgtPooki commented 1 year ago

re-arranged priority of projects according to https://github.com/ipfs-shipyard/ignite-metrics/issues/32#issuecomment-1386173775, and removed ipfs-share-files

cc @tinytb

SgtPooki commented 1 year ago

In order to get metrics applied to our non-vital projects, I'm going to move forward with enabling the metrics and creating a backlog task for adding the UI treatment (toggle UI) to those projects. This will help us prioritize getting the numbers we need, while still allowing users to opt out (instructions should be provided in the backlog tacks for adding the toggles).

SgtPooki commented 1 year ago

PR is out for adding telemetry to starmap project: https://github.com/pln-planning-tools/Starmap/pull/272, but dependent upon some updates to ignite-metrics

SgtPooki commented 1 year ago

Status update:

A Dashboard with existing metrics can be found at https://pl-strflt.notion.site/IPFS-Ignite-Metrics-Dashboard-dee2ada6225e4e01a172673496851210

There is a callouts section on the dashboard page to indicate the current state, but here is a recap:

  1. We are now collecting telemetry for all of our projects except ipfs-companion
  2. ipfs-companion metrics will be added once the following is done:
  3. ipfs-desktop and ipfs-webui will have switched to opt-out metrics once the following is done:

This comment will also be added into the description

SgtPooki commented 1 year ago

:mega: All IPFS Ignite team projects have opt-out metrics added, and are released.

We’re still waiting on updates to ipfs-companion in the chrome store, but ipfs-companion has a new release: https://github.com/ipfs/ipfs-companion/releases/tag/v2.21.0 and is currently available via firefox.

ipfs-webui has been released - https://github.com/ipfs/ipfs-webui/releases/tag/v2.22.0 ipfs-desktop has been released - https://github.com/ipfs/ipfs-desktop/releases/tag/v0.26.0 with the new webui version.

we're waiting on Kubo to merge update of latest webui version: https://github.com/ipfs/kubo/pull/9597

What metrics don't we have yet?

When will we have all of our metrics?

cc @tinytb @biglep

SgtPooki commented 1 year ago

status update:

SgtPooki commented 1 year ago

marking this as blocked because the only remaining action item is to confirm that the ipfs-webui-kubo metrics are showing properly once a kubo release is made: https://github.com/ipfs/kubo/issues/9502

SgtPooki commented 1 year ago

we're getting metrics from kubo webui now: https://countly.ipfs.tech/dashboard#/63c596762a7760344a6b2cfd/. Something is up with metrics in countly though.. when I just logged in they prompted me for 2FA, so i'm assuming the authentication for the robot client needs something as well

SgtPooki commented 1 year ago

image

kubo metrics have been pretty stable, and is a higher usecase than webui directly. closing this since all of our projects have metrics.