search

hypothesis / vision

Envisioning the future of the Hypothesis.
https://github.com/hypothesis/vision/issues/
40 stars 8 forks source link

Metrics / Statistics #75

Closed dwhly closed 9 years ago

dwhly commented 10 years ago

When running a service, it's useful to know some key metrics that characterize the major dimensions of activity.

These are an initial set of simple metrics. These should be gathered and reported daily via an email to team@hypothes.is.

Acceptance criteria for the initial set:

Please include any additional metrics from the list below that are easy to gather— or we can backlog these:

This evolved from an earlier request, preserved below:

"Recently, U Mich requested / asked for access to the following info as a partner:

tilgovi commented 10 years ago

Great issue to have around.

This is one of those places where our extension differs from our partnerships around product. For the extension, metrics are good for our use in order to make data-driven decisions, but it's a user facing feature for the "I'm a partner and..." class of stories.

You'd think there might not be a difference if we treat ourselves as a partner in this sense (@BigBlueHat tip), but the difference is that if we make it a feature facing external users we also need to invest in its usability and aesthetic whereas internal-only dashboards can be hella ugly and still accomplish their purpose.

BigBlueHat commented 10 years ago

Would love to have these metrics, but think an actual "dashboard" should be considered out of scope for this issue.

First hard task is making them available in JSON / CSV.

Next one is caching the tar out of these so we're not re-querying the entire database every time someone feels like accessing the metrics.

Anything beyond that is a product unto itself with it's own life cycle, needs, demands, etc. In fact, it should be a separate product that consumes our stats--if/when we find it valuable for any of our partners (including the "general public" one).

Sound about right?

aron commented 10 years ago

+1 on this being purely for the public service at the moment.

First hard task is making them available in JSON / CSV.

Just putting it out there but there are plenty of external services (both OS and paid) that do this, should we not want to spend time on building infrastructure. keen.io is a paid one I used recently that's pretty good.

BigBlueHat commented 10 years ago

I actually have local friends who work at keen.io and was looking at their stuff this morning re: this issue. :smile:

Worth a look. Might be enough for us to get started with at least for now.

:+1: for exploring keen.io

On Thu, Aug 14, 2014 at 10:39 AM, Aron Carroll notifications@github.com wrote:

+1 on this being purely for the public service at the moment.

First hard task is making them available in JSON / CSV.

Just putting it out there but there are plenty of external services (both OS and paid) that do this, should we not want to spend time on building infrastructure. keen.io is a paid one I used recently that's pretty good.

— Reply to this email directly or view it on GitHub https://github.com/hypothesis/vision/issues/75#issuecomment-52191543.

dwhly commented 10 years ago

Happy to consider the API a different deliverable than a dashboard. But should I create a separate issue for how to provide this for customers, since that's what drove the original rationale? Doesn't have to be immediate term--- but I do want an issue to point to for when folks request this, such as U Mich. I guess what I'm wondering is: Will some customers like W3C have an issue w/ sending data to a 3rd party to get basic details about transactions?

tilgovi commented 10 years ago

I also know people at keen. Would be up for trying it out at some point. On Aug 14, 2014 7:40 AM, "BigBlueHat" notifications@github.com wrote:

I actually have local friends who work at keen.io and was looking at their stuff this morning re: this issue. :smile:

Worth a look. Might be enough for us to get started with at least for now.

:+1: for exploring keen.io

On Thu, Aug 14, 2014 at 10:39 AM, Aron Carroll notifications@github.com wrote:

+1 on this being purely for the public service at the moment.

First hard task is making them available in JSON / CSV.

Just putting it out there but there are plenty of external services (both OS and paid) that do this, should we not want to spend time on building infrastructure. keen.io is a paid one I used recently that's pretty good.

— Reply to this email directly or view it on GitHub https://github.com/hypothesis/vision/issues/75#issuecomment-52191543.

— Reply to this email directly or view it on GitHub https://github.com/hypothesis/vision/issues/75#issuecomment-52191763.

BigBlueHat commented 9 years ago

From the original list, but grouped by data source.

SQL

NOTE: It's not clear if we're collecting (or storing) enough data related to activation long enough to generate some of these numbers.

ElasticSearch
nickstenning commented 9 years ago

Closing this in favour of split out cards.

dwhly commented 9 years ago

This is now referenced under #151